Domain adaptive speech recognition method and related device

Abstract

The invention discloses a domain adaptive speech recognition method and a related device, and relates to the technical field of speech processing, and the method comprises the steps: obtaining a domain adaptive speech recognition model through the training of target domain unsupervised data in advance, and obtaining a target domain dictionary when the to-be-recognized target domain speech needs to be recognized, the target domain voice and the target domain dictionary are input into the domain self-adaptive voice recognition model, the domain self-adaptive voice recognition model recognizes the target domain voice based on the target domain dictionary, and a recognition result is obtained. According to the scheme, the target domain unsupervised data is utilized in the training stage, the target domain dictionary is utilized in the prediction stage, and the recognition efficiency is improved. Therefore, the speech recognition effect of the target field can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a domain-adaptive speech recognition method and related devices. Background Art

[0002] In the early stages of developing emerging business scenarios, the application of speech recognition algorithms is often limited by the scarcity of data in the target field, which affects the recognition effect and the smooth implementation of the business. Taking the field of geological exploration as an example, the application of intelligent dialogue systems faces many challenges. The high intensity of background noise, the use of professional terms by users, accent differences, and changes in speaking distance all have a significant impact on the performance of automatic speech recognition systems. In particular, when the system has not been exposed to accurately labeled geological exploration scene data, its performance is particularly degraded. In view of this, it is crucial to develop a speech recognition algorithm that can quickly adapt to new fields.

[0003] Existing domain-adaptive speech recognition algorithms do not effectively utilize the unsupervised data in the target domain, resulting in poor recognition results.

[0004] Therefore, how to provide a domain-adaptive speech recognition algorithm to fully utilize the unsupervised data in the target domain and thus improve the recognition effect has become a technical problem that needs to be urgently solved by technicians in this field. Summary of the invention

[0005] In view of the above problems, this application provides a domain-adaptive speech recognition method and related devices to achieve the purpose of making full use of unsupervised data in the target domain and thus improving the recognition effect. The specific scheme is as follows:

[0006] The first aspect of the present application provides a domain adaptive speech recognition method, comprising:

[0007] Obtaining the target domain speech to be recognized and the target domain dictionary;

[0008] The target domain speech and the target domain dictionary are input into a domain adaptive speech recognition model, and the domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result; the domain adaptive speech recognition model is trained using unsupervised data of the target domain.

[0009] In a possible implementation, the target domain unsupervised data includes target domain background noise and / or target domain text.

[0010] In a possible implementation, the training method of the domain adaptive speech recognition model includes:

[0011] Acquire a non-target domain speech sample and a recognition text label of the non-target domain speech sample;

[0012] Get pre-built domain-adaptive speech recognition models;

[0013] The domain adaptive speech recognition model is trained by multi-task joint training using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to obtain a trained domain adaptive speech recognition model.

[0014] In a possible implementation, when the target domain unsupervised data includes target domain background noise and target domain text, the multi-task joint training of the domain adaptive speech recognition model using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data includes:

[0015] The non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data are used to perform joint training of a noisy speech recognition task, a domain background noise adaptation task and a domain text mask prediction task.

[0016] In a possible implementation, the domain adaptive speech recognition model includes an encoder and a decoder; then the use of the non-target domain speech samples, the recognition text labels of the non-target domain speech samples, and the target domain unsupervised data to perform joint training of noisy speech recognition tasks, domain background noise adaptation tasks, and domain text mask prediction tasks includes:

[0017] Input the non-target domain speech sample and the target domain background noise into the initial layer of the encoder, and input the target domain text into the middle layer of the encoder after masking;

[0018] The loss function of the noisy speech recognition task, the loss function of the domain background noise adaptation task, and the loss function of the domain text mask prediction task are constructed as a joint loss function;

[0019] The domain adaptive speech recognition model is trained based on the joint loss function.

[0020] In a possible implementation, the domain-adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result, including:

[0021] The domain adaptive speech recognition model determines the prediction weight of each word in a preset decoding dictionary at each decoding step;

[0022] Determine a target word from the preset decoding dictionary; the target word is a word in the preset decoding dictionary whose similarity with a word in the target domain dictionary meets a preset condition;

[0023] Weighting the prediction weights of the target words to obtain final prediction weights of each word in the preset decoding dictionary;

[0024] Based on the final prediction weight of each word in the preset decoding dictionary, the recognition result of the decoding step is determined.

[0025] A second aspect of the present application provides a domain adaptive speech recognition device, comprising:

[0026] An acquisition unit, used to acquire the target domain speech to be recognized and the target domain dictionary;

[0027] The recognition unit is used to input the target domain speech and the target domain dictionary into a domain adaptive speech recognition model, and the domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result; the domain adaptive speech recognition model is trained using unsupervised data in the target domain.

[0028] In a possible implementation, the target domain unsupervised data includes target domain background noise and / or target domain text.

[0029] In a possible implementation, the apparatus further includes: a domain adaptive speech recognition model training unit;

[0030] The domain adaptive speech recognition model training unit specifically includes:

[0031] A training data acquisition unit, used to acquire non-target domain speech samples and recognition text labels of the non-target domain speech samples;

[0032] A model acquisition unit, used to acquire a pre-built domain adaptive speech recognition model;

[0033] The training unit is used to perform multi-task joint training on the domain adaptive speech recognition model by using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to obtain a trained domain adaptive speech recognition model.

[0034] In a possible implementation, when the target domain unsupervised data includes target domain background noise and target domain text, the training unit includes:

[0035] A multi-task joint training unit is used to use the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to perform joint training of noisy speech recognition tasks, domain background noise adaptation tasks and domain text mask prediction tasks.

[0036] In a possible implementation, the domain adaptive speech recognition model includes an encoder and a decoder; then the multi-task joint training unit is specifically used to:

[0037] Input the non-target domain speech sample and the target domain background noise into the initial layer of the encoder, and input the target domain text into the middle layer of the encoder after masking;

[0038] The loss function of the noisy speech recognition task, the loss function of the domain background noise adaptation task, and the loss function of the domain text mask prediction task are constructed as a joint loss function;

[0039] The domain adaptive speech recognition model is trained based on the joint loss function.

[0040] In a possible implementation, the identification unit is specifically configured to:

[0041] The domain adaptive speech recognition model determines the prediction weight of each word in a preset decoding dictionary at each decoding step;

[0042] Determine a target word from the preset decoding dictionary; the target word is a word in the preset decoding dictionary whose similarity with a word in the target domain dictionary meets a preset condition;

[0043] Weighting the prediction weights of the target words to obtain final prediction weights of each word in the preset decoding dictionary;

[0044] Based on the final prediction weight of each word in the preset decoding dictionary, the recognition result of the decoding step is determined.

[0045] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the domain-adaptive speech recognition method of the first aspect or any implementation method of the first aspect.

[0046] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program so that the electronic device can implement the domain adaptive speech recognition method of the above-mentioned first aspect or any implementation method of the first aspect.

[0049] A fifth aspect of the present application provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the domain-adaptive speech recognition method of the above-mentioned first aspect or any implementation of the first aspect.

[0050] By means of the above-mentioned technical scheme, the domain adaptive speech recognition method and related devices provided by the present application pre-train a domain adaptive speech recognition model using unsupervised data of the target domain. When it is necessary to recognize the target domain speech to be recognized, the target domain dictionary is first obtained, and the target domain speech and the target domain dictionary are input into the domain adaptive speech recognition model. The domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result. This scheme utilizes the unsupervised data of the target domain in the training stage and utilizes the target domain dictionary in the prediction stage. Therefore, the effect of speech recognition in the target domain can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0052] Figure 1 A flowchart of a domain-adaptive speech recognition method provided in an embodiment of the present application;

[0053] Figure 2 A flowchart of a method for training a domain-adaptive speech recognition model provided in an embodiment of the present application;

[0054] Figure 3 A flowchart of a joint training of a noisy speech recognition task, a domain background noise adaptation task, and a domain text mask prediction task using non-target domain speech samples, recognition text labels of non-target domain speech samples, and target domain unsupervised data disclosed in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of a process for obtaining a recognition result by recognizing a target domain speech based on a target domain dictionary using a domain adaptive speech recognition model provided in an embodiment of the present application;

[0056] Figure 5A schematic diagram of the structure of a domain-adaptive speech recognition device provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0059] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0060] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0061] In the early stages of developing emerging business scenarios, the application of speech recognition algorithms is often limited by the scarcity of data in the target field, which affects the recognition effect and the smooth implementation of the business. Taking the field of geological exploration as an example, the application of intelligent dialogue systems faces many challenges. The high intensity of background noise, the use of professional terms by users, accent differences, and changes in speaking distance all have a significant impact on the performance of automatic speech recognition systems. In particular, when the system has not been exposed to accurately labeled geological exploration scene data, its performance is particularly degraded. In view of this, it is crucial to develop an unsupervised speech recognition algorithm that can quickly adapt to new fields.

[0062] There are two main types of existing unsupervised speech recognition algorithms. One is to use a batch of data for joint training by manual rapid labeling. This type of solution first requires professional labelers to perform manual labeling, and it also takes a lot of time. The other is to use a joint domain dictionary for speech recognition. On the one hand, this method does not take into account the impact of background sound in the target domain. On the other hand, because the model does not learn the text style of the target domain, it will lead to serious misrecognition.

[0063] For the domain joint dictionary speech recognition method, one is the cascade architecture based on HMM-GMM+ language model. The advantage of this type of architecture is the flexible customization of domain dictionaries and simple structure, but the disadvantage is poor effect. The other is the end-to-end architecture based on ED. The disadvantage of this type of solution is that dictionary customization is not flexible enough, domain dictionaries need to be continuously iterated and replaced, and the utilization of unsupervised data is low.

[0064] It can be seen that the existing unsupervised speech recognition algorithms do not effectively make full use of the unsupervised data in the target field, resulting in poor recognition results.

[0065] In order to solve the above problems, the embodiment of the present application provides a domain adaptive speech recognition method. The domain adaptive speech recognition method of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0066] Reference Figure 1 , Figure 1 A flowchart of a domain adaptive speech recognition method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, a domain adaptive speech recognition method provided by an embodiment of the present application may include the following steps, which are described in detail below.

[0067] S101: Acquire target domain speech to be recognized and target domain dictionary;

[0068] In this application, the target field can be any field, such as geological exploration field, medical field, etc., and this application does not impose any limitation on this. The target field dictionary contains commonly used words in the target field (such as professional terms, etc.). Taking the geological exploration field as an example, the target field dictionary can contain words such as "drilling rig", "drilling well", and "drilling point".

[0069] It should be noted that the target domain dictionary can be updated to include the latest and most complete domain terms, thereby improving the speech recognition effect in the target domain.

[0070] S102: Input the target domain speech and the target domain dictionary into a domain adaptive speech recognition model, wherein the domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result; the domain adaptive speech recognition model is trained using unsupervised data from the target domain.

[0071] From the perspective of data acquisition, unsupervised data in the target domain (such as background noise, text materials and professional dictionaries in the field of geological exploration) can be collected efficiently in a short time, and the time cost involved is negligible. Therefore, in this application, the domain-adaptive speech recognition model can be trained using unsupervised data in the target domain.

[0072] The domain-adaptive speech recognition model obtained by training with unsupervised data in the target domain does not require additional human and material resources from the perspective of economic cost, and can achieve highly automated processing. In addition, the domain-adaptive speech recognition model obtained by training with unsupervised data in the target domain enables the model to learn the unique semantics of the target domain and improve the effect of speech recognition in the target domain. Therefore, the domain-adaptive speech recognition model obtained by training with unsupervised data in the target domain can assist related companies in quickly building speech recognition models that are friendly to the target domain, thereby promoting the rapid implementation of related businesses.

[0073] In the present application, the target domain unsupervised data includes target domain background noise, and / or target domain text. In the present application, before the target domain speech and the target domain dictionary are input into the domain adaptive speech recognition model, the target domain speech can be firstly feature extracted to obtain the features of the target domain speech; and the target domain dictionary can be feature extracted to obtain the target domain dictionary representation; and then the features of the target domain speech and the target domain dictionary representation are input into the domain adaptive speech recognition model.

[0074] In the present application, the feature of the target domain speech may be an Fbank feature. In a possible implementation, the waveform A corresponding to the target domain speech may be extracted using the librosa library. F , the sampling rate is 16000, and then the frame splitting and windowing strategy is used to obtain the Fbank feature A corresponding to the waveform E .

[0075] In this application, the BERT model can be used to extract features from the target domain dictionary to obtain the target domain dictionary representation. For ease of description, in this application, V E Represents the target domain dictionary representation.

[0076] The domain adaptive speech recognition method provided in this embodiment uses the target domain unsupervised data to train the domain adaptive speech recognition model in advance. When the target domain speech needs to be recognized, the target domain dictionary is first obtained, and the target domain speech and the target domain dictionary are input into the domain adaptive speech recognition model. The domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result. This solution uses the target domain unsupervised data in the training stage and uses the target domain dictionary in the prediction stage. Therefore, the effect of target domain speech recognition can be improved.

[0077] In another embodiment of the present application, the training method of the domain adaptive speech recognition model is described in detail.

[0078] Reference Figure 2 , Figure 2 A flowchart of a method for training a domain-adaptive speech recognition model provided in an embodiment of the present application is provided. The method may include the following steps:

[0079] S201: Acquire a non-target domain speech sample and a recognition text label of the non-target domain speech sample;

[0080] S202: Obtain a pre-built domain-adaptive speech recognition model;

[0081] In the present application, the domain adaptive speech recognition model includes an encoder and a decoder; that is, the architecture of the domain adaptive speech recognition model can be an encoder-decoder structure. For example, the architecture of the domain adaptive speech recognition model can adopt an ED (Encoder-Decoder) network based on Conformer. For another example, the architecture of the domain adaptive speech recognition model can be an ED (Encoder-Decoder) network based on Transformer.

[0082] S203: Perform multi-task joint training on the domain adaptive speech recognition model by using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to obtain a trained domain adaptive speech recognition model.

[0083] In a possible implementation, when the target domain unsupervised data includes target domain background noise and target domain text, the multi-task joint training of the domain adaptive speech recognition model using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data includes:

[0084] The non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data are used to perform joint training of a noisy speech recognition task, a domain background noise adaptation task and a domain text mask prediction task.

[0085] In another embodiment of the present application, a specific implementation method for performing joint training of noisy speech recognition tasks, domain background noise adaptation tasks, and domain text mask prediction tasks using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples, and the target domain unsupervised data is described.

[0086] Reference Figure 3 , Figure 3The present invention discloses a flowchart of a joint training of a noisy speech recognition task, a domain background noise adaptation task, and a domain text mask prediction task using non-target domain speech samples, recognition text labels of non-target domain speech samples, and target domain unsupervised data, which specifically includes the following steps:

[0087] S301: Input the non-target domain speech sample and the target domain background noise into the initial layer of the encoder, and input the target domain text into the middle layer of the encoder after masking;

[0088] In the present application, the non-target domain speech sample and the target domain background noise are input into the initial layer of the encoder, and the target domain text is masked and input into the middle layer of the encoder, including: extracting features of the non-target domain speech sample to obtain features of the non-target domain speech sample; extracting features of the target domain background noise to obtain features of the target domain background noise; filtering and dimension mapping the target domain text to obtain a target domain text representation; masking the target domain text representation to obtain a masked target domain text representation; inputting the features of the non-target domain speech sample and the target domain background noise features into the initial layer of the encoder, and inputting the masked target domain text representation into the middle layer of the encoder.

[0089] It should be noted that, in the present application, the features of the non-target domain speech samples may be Fbank features. The extraction method thereof may refer to the extraction method of the features of the target domain speech to be identified, and the description thereof will not be repeated here. The target domain background noise features may be Fbank features of the target domain background noise. The extraction method thereof may refer to the extraction method of the features of the target domain audio to be identified, and the description thereof will not be repeated here. For ease of description, in the present application, N E In this application, invalid text and harmful text in the target domain text can be filtered, and the existing dimension mapping module can be used to perform dimension mapping processing on the filtered target domain text to obtain the target domain text representation. For ease of description, in this application, T E Represents the target domain text representation.

[0090] It should also be noted that, in the present application, the target domain text is masked by masking any word, and preferably, the words in the target domain text that are highly relevant to the domain are masked. To facilitate understanding of the masking of the target domain text, an example of the target domain text before and after masking is given in the present application, as follows:

[0091] Before masking: "The lithology here is suitable for drilling"

[0092] After mask processing: "×× here is suitable for drilling"

[0093] It is worth noting that the target domain text can be input into any intermediate layer of the encoder after being masked, which can be determined based on scenario requirements and is not limited in this application.

[0094] In this application, inserting masked target domain text into the middle layer of the model's encoder allows the model to try to predict the masked parts during the learning process, thereby further improving its recognition accuracy in specific fields.

[0095] S302: constructing a joint loss function from the loss function of the noisy speech recognition task, the loss function of the domain background noise adaptation task, and the loss function of the domain text mask prediction task;

[0096] In the present application, the non-target domain speech sample and the target domain background noise are input into the initial layer of the encoder, and the target domain background noise is directly superimposed on the non-target domain speech sample. In this way, the model can learn and eventually generate a speech prediction result with noise (Prediction noise). The non-target domain speech sample can also be input into the initial layer of the encoder to obtain a speech prediction result without noise (Predictionclean). Using the speech prediction result without noise (Prediction clean) to guide the speech prediction result with noise (Prediction noise) can enable the model to have the ability to adapt to the domain background noise.

[0097] In the present application, after the target domain text is masked and input into the middle layer of the encoder, the model generates a mask prediction result.

[0098] In the present application, the loss function of the noisy speech recognition task is used to indicate the cross entropy loss between the noisy speech prediction result (Prediction noise) generated by the model and the recognition text label of the non-target domain speech sample.

[0099] In the present application, the loss function of the domain background noise adaptation task is used to indicate the KL divergence between the speech prediction results with noise (Prediction noise) and the speech prediction results without noise (Prediction clean) generated by the model.

[0100] For ease of understanding, assume that the domain adaptive speech recognition model is represented by F and the non-target domain speech sample is represented by AE , the background noise in the target area is N E , the prediction result of speech with noise (Prediction noise) is P C , the prediction result of speech without noise (Prediction clean) is P N , then the loss function Loss of the domain background noise adaptation task is kl It can be expressed as:

[0101]

[0102]

[0103]

[0104] In the present application, the loss function of the domain text mask prediction task is used to indicate the difference between the mask prediction result and the text label of the mask position.

[0105] For ease of understanding, assume that the domain adaptive speech recognition model is represented by F and the target domain text is represented by T E , the target domain text is masked to M E , the mask prediction result is P T , then the loss function of the domain text mask prediction task can be expressed as:

[0106]

[0107]

[0108]

[0109] S303: Training the domain adaptive speech recognition model based on the joint loss function.

[0110] In the present application, after the joint loss function converges, the training of the domain adaptive speech recognition model is completed.

[0111] In another embodiment of the present application, a specific implementation method of obtaining a recognition result by recognizing the target domain speech based on the target domain dictionary by the domain adaptive speech recognition model is described.

[0112] Reference Figure 4 , Figure 4 A flow chart of a domain-adaptive speech recognition model provided in an embodiment of the present application for recognizing speech in a target domain based on a target domain dictionary to obtain a recognition result specifically includes the following steps:

[0113] S401: The domain adaptive speech recognition model determines the prediction weight of each word in a preset decoding dictionary at each decoding step;

[0114] In the present application, the domain adaptive speech recognition model can determine the prediction weight of each word in the preset decoding dictionary at each decoding step; the specific implementation method is the existing technology and will not be elaborated in this application.

[0115] S402: Determine a target word from the preset decoding dictionary; the target word is a word in the preset decoding dictionary whose similarity with a word in the target domain dictionary meets a preset condition;

[0116] In the present application, after determining the prediction weight of each word in the preset decoding dictionary, for each word, its similarity with each word in the target domain dictionary can be calculated, and if the similarity meets the preset condition, the word is determined to be the target word. In a possible implementation, a similarity threshold (such as 0.5) can be preset, and if the similarity is greater than the preset threshold, it is determined that the similarity meets the preset condition.

[0117] S403: weighting the prediction weight of the target word to obtain the final prediction weight of each word in the preset decoding dictionary;

[0118] After weighting the prediction weights of the target words, the final prediction weights of the words in the preset decoding dictionary are obtained, and the vocabulary weights of the target domain will be increased accordingly.

[0119] S404: Determine the recognition result of the decoding step based on the final prediction weight of each word in the preset decoding dictionary.

[0120] By weighting the prediction weight of the target word, the final prediction weight of each word in the preset decoding dictionary is obtained, and the vocabulary weight of the target field has been correspondingly improved. Then, when determining the recognition result of the decoding step based on the final prediction weight of each word in the preset decoding dictionary, it can ensure that the vocabulary in the target field is accurately recognized and transcribed, effectively improving the model's recognition accuracy for key vocabulary in the target field.

[0121] In a preferred implementation, the target domain dictionary can be divided based on the usage frequency of each word in the target domain dictionary at different decoding stages. At the decoding step of different decoding stages, the part of the target domain dictionary corresponding to the decoding stage is used to determine the target words, so as to further ensure that the vocabulary of the target domain can be accurately identified and transcribed, thereby further improving the recognition accuracy of the model for key vocabulary in the target domain.

[0122] In summary, this solution integrates the background noise and text data of the target domain into the training phase of the model. Even in the absence of supervised speech recognition data in the target domain, it can quickly achieve adaptive speech recognition in the target domain, thereby accelerating the effective deployment of relevant speech recognition services in the target domain.

[0123] This solution can be widely used in speech recognition tasks in specific fields under various new business scenarios. Its notable feature is that even in the absence of supervised speech recognition training data in a specific field, it can quickly and accurately recognize the speech in the target field by utilizing one or more unsupervised data in the target field.

[0124] The above introduces a domain adaptive speech recognition method provided by an embodiment of the present application. The following will introduce a device for executing the above domain adaptive speech recognition method.

[0125] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a domain adaptive speech recognition device provided in an embodiment of the present application. Figure 5 As shown, the field adaptive speech recognition device comprises:

[0126] An acquisition unit 11 is used to acquire the target domain speech to be recognized and the target domain dictionary;

[0127] The recognition unit 12 is used to input the target domain speech and the target domain dictionary into a domain adaptive speech recognition model, and the domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result; the domain adaptive speech recognition model is trained using unsupervised data in the target domain.

[0128] In a possible implementation, the target domain unsupervised data includes target domain background noise and / or target domain text.

[0129] In a possible implementation, the apparatus further includes: a domain adaptive speech recognition model training unit;

[0130] The domain adaptive speech recognition model training unit specifically includes:

[0131] A training data acquisition unit, used to acquire non-target domain speech samples and recognition text labels of the non-target domain speech samples;

[0132] A model acquisition unit, used to acquire a pre-built domain adaptive speech recognition model;

[0133] The training unit is used to perform multi-task joint training on the domain adaptive speech recognition model by using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to obtain a trained domain adaptive speech recognition model.

[0134] In a possible implementation, when the target domain unsupervised data includes target domain background noise and target domain text, the training unit includes:

[0135] A multi-task joint training unit is used to use the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to perform joint training of noisy speech recognition tasks, domain background noise adaptation tasks and domain text mask prediction tasks.

[0136] In a possible implementation, the domain adaptive speech recognition model includes an encoder and a decoder; then the multi-task joint training unit is specifically used to:

[0137] Input the non-target domain speech sample and the target domain background noise into the initial layer of the encoder, and input the target domain text into the middle layer of the encoder after masking;

[0138] The loss function of the noisy speech recognition task, the loss function of the domain background noise adaptation task, and the loss function of the domain text mask prediction task are constructed as a joint loss function;

[0139] The domain adaptive speech recognition model is trained based on the joint loss function.

[0140] In a possible implementation, the identification unit is specifically configured to:

[0141] The domain adaptive speech recognition model determines the prediction weight of each word in a preset decoding dictionary at each decoding step;

[0142] Determine a target word from the preset decoding dictionary; the target word is a word in the preset decoding dictionary whose similarity with a word in the target domain dictionary meets a preset condition;

[0143] Weighting the prediction weights of the target words to obtain final prediction weights of each word in the preset decoding dictionary;

[0144] Based on the final prediction weight of each word in the preset decoding dictionary, the recognition result of the decoding step is determined.

[0145] The present application also provides an electronic device in an embodiment. Figure 6As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0146] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0148] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any domain-adaptive speech recognition method provided in the embodiment of the present application.

[0149] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any domain-adaptive speech recognition method provided in an embodiment of the present application.

[0150] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0151] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0152] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0153] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A domain adaptive speech recognition method, characterized in that: include: Obtaining the target domain speech to be recognized and the target domain dictionary; The target domain speech and the target domain dictionary are input into a domain adaptive speech recognition model, and the domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result; the domain adaptive speech recognition model is trained using unsupervised data of the target domain.

2. The method according to claim 1, characterized in that The target domain unsupervised data includes target domain background noise and / or target domain text.

3. The method according to claim 2, characterized in that The training method of the domain adaptive speech recognition model includes: Acquire a non-target domain speech sample and a recognition text label of the non-target domain speech sample; Get pre-built domain-adaptive speech recognition models; The domain adaptive speech recognition model is trained by multi-task joint training using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to obtain a trained domain adaptive speech recognition model.

4. The method according to claim 3, characterized in that: When the target domain unsupervised data includes target domain background noise and target domain text, the multi-task joint training of the domain adaptive speech recognition model using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data includes: The non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data are used to perform joint training of a noisy speech recognition task, a domain background noise adaptation task and a domain text mask prediction task.

5. The method according to claim 4, characterized in that The domain adaptive speech recognition model includes an encoder and a decoder; the method of using the non-target domain speech samples, the recognition text labels of the non-target domain speech samples and the target domain unsupervised data to perform joint training of noisy speech recognition tasks, domain background noise adaptation tasks and domain text mask prediction tasks includes: Input the non-target domain speech sample and the target domain background noise into the initial layer of the encoder, and input the target domain text into the middle layer of the encoder after masking; The loss function of the noisy speech recognition task, the loss function of the domain background noise adaptation task, and the loss function of the domain text mask prediction task are constructed as a joint loss function; The domain adaptive speech recognition model is trained based on the joint loss function.

6. The method according to claim 1, characterized in that The domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result, including: The domain adaptive speech recognition model determines the prediction weight of each word in a preset decoding dictionary at each decoding step; Determine a target word from the preset decoding dictionary; the target word is a word in the preset decoding dictionary whose similarity with a word in the target domain dictionary meets a preset condition; Weighting the prediction weights of the target words to obtain final prediction weights of each word in the preset decoding dictionary; Based on the final prediction weight of each word in the preset decoding dictionary, the recognition result of the decoding step is determined.

7. A domain-adaptive speech recognition device, characterized in that: include: An acquisition unit, used to acquire the target domain speech to be recognized and the target domain dictionary; The recognition unit is used to input the target domain speech and the target domain dictionary into a domain adaptive speech recognition model, and the domain adaptive speech recognition model recognizes the target domain speech based on the target domain dictionary to obtain a recognition result; the domain adaptive speech recognition model is trained using unsupervised data in the target domain.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the domain-adaptive speech recognition method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the domain adaptive speech recognition method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the domain adaptive speech recognition method as described in any one of claims 1 to 6.

Images