Speech recognition method and model training method

Abstract

The application discloses a speech recognition method and a model training method. The method comprises the following steps: obtaining speech data obtained by collecting speech of a first object, wherein the speech data contains speech content of a target language type; extracting features of the speech data by using a speech recognition model to obtain acoustic features of the speech data; inputting the acoustic features into the speech recognition model based on global information and local information of the acoustic features to perform speech recognition, and obtaining a target recognition result of the speech content, wherein the speech recognition model is obtained by performing semi-supervised and self-supervised learning on first speech samples and second speech samples, the first speech samples contain speech samples of multiple preset language types and do not contain labeled information, and the second speech samples contain language samples of the target language type and labeled information. The application solves the technical problem of low recognition accuracy of a speech recognition model in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of semantic recognition, and in particular, to a speech recognition method and a model training method. BACKGROUND

[0002] At present, in the field of speech recognition, traditional multi-language speech recognition needs to rely on a large number of annotated speech data of different scenes and different languages to improve the recognition ability of the speech recognition model. However, in addition to the widely used languages, the current multi-language is generally a low-resource language, that is, it generally contains fewer resource samples, resulting in poor training effect of the speech recognition model, thereby causing poor speech recognition ability of the speech recognition model for multi-language.

[0003] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0004] Embodiments of the present application provide a speech recognition method and a model training method to at least solve the technical problem of low recognition accuracy of the speech recognition model in the related art.

[0005] According to an aspect of an embodiment of the present application, a speech recognition method is provided, comprising: obtaining speech data obtained by speech collection on a first object, wherein the speech data contains speech content of a target language type, and the target language type is any one of a plurality of preset language types; performing feature extraction on the speech data by using a speech recognition model to obtain acoustic features of the speech data; inputting the acoustic features into the speech recognition model based on global information and local information of the acoustic features to perform speech recognition, and obtaining a target recognition result of the speech content, wherein the speech recognition model is obtained by semi-supervised and self-supervised learning using first speech samples and second speech samples, the first speech samples contain speech samples of a plurality of preset language types and do not contain annotation information, and the second speech samples contain language samples of the target language type and annotation information.

[0006] According to an aspect of an embodiment of the present application, a model training method is provided, comprising: obtaining first speech samples and second speech samples, wherein the first speech samples contain speech samples of a plurality of preset language types and do not contain annotation information, the second speech samples contain language samples of a target language type and annotation information, and the target language type is any one of the plurality of preset language types; performing semi-supervised and self-supervised learning on an initial recognition model using the first speech samples and the second speech samples to obtain a speech recognition model, wherein the speech recognition model is used to perform speech recognition on speech content contained in speech data based on global information and local information of the speech data to obtain a target recognition result of the speech content, and the language type corresponding to the speech content is the target language type.

[0007] According to an aspect of the embodiments of the present application, a voice recognition method is provided, comprising: in response to an input instruction acting on an operation interface, driving a voice collection device to collect voice of a first object to obtain voice data, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; in response to a voice recognition instruction acting on the operation interface, displaying a target recognition result of the voice content on the operation interface, wherein the target recognition result is obtained by inputting acoustic features of the voice data into a voice recognition model for voice recognition based on global information and local information of the acoustic features, the acoustic features are obtained by using the voice recognition model to extract features of the voice data, and the voice recognition model is obtained by using a first voice sample and a second voice sample for semi-supervised and self-supervised learning, the first voice sample contains voice samples of the plurality of preset language types and does not contain labeled information, and the second voice sample contains language samples of the target language type and labeled information.

[0008] According to an aspect of the embodiments of the present application, a voice recognition method is provided, comprising: obtaining voice data obtained by collecting voice of a first object by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the voice data, the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; extracting features of the voice data by using a voice recognition model to obtain acoustic features of the voice data; inputting the acoustic features into the voice recognition model for voice recognition based on global information and local information of the acoustic features to obtain a target recognition result of the voice content, wherein the voice recognition model is obtained by using a first voice sample and a second voice sample for semi-supervised and self-supervised learning, the first voice sample contains voice samples of the plurality of preset language types and does not contain labeled information, and the second voice sample contains language samples of the target language type and labeled information; and outputting the target recognition result by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target recognition result.

[0009] According to an aspect of the embodiments of the present application, a computer readable storage medium is provided, which includes a stored program, wherein the program controls a device where the computer readable storage medium is located to execute the method of any one of the above embodiments when the program is running.

[0010] According to an aspect of the embodiments of the present application, a computer terminal is provided, comprising: a memory storing an executable program; and a processor configured to run the program, wherein the program controls the computer terminal to execute the method of any one of the above embodiments when the program is running.

[0011] By the above steps, the voice data obtained by voice collection on the first object is acquired, wherein the voice data contains voice content of a target language type, and the target language type is any one of multiple preset language types; acoustic features of the voice data are obtained by using a voice recognition model to perform feature extraction on the voice data; and the target recognition result of the voice content is obtained by inputting the acoustic features into the voice recognition model for voice recognition based on global information and local information of the acoustic features, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of multiple preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information, thereby improving the accuracy of voice data recognition; in this application, the voice recognition model can be trained by using multi-task semi-supervised and self-supervised learning, which can improve the generalization ability of the voice recognition model, and the voice recognition model can also perform high-precision recognition on voice content of a language type with a small number of samples, thereby improving the recognition accuracy of voice data and solving the technical problem of low recognition accuracy of the voice recognition model in related technologies.

[0012] It is easy to note that the general description above and the detailed description below are merely examples and explanations of the present application and do not constitute limitations on the present application. BRIEF DESCRIPTION OF DRAWINGS

[0013] The drawings described herein are used to provide further understanding of the present application, constitute a part of the present application, the illustrative embodiments of the present application and the description thereof are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0014] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a voice recognition method according to an embodiment of the present application;

[0015] Figure 2 is a flowchart of a voice recognition method according to embodiment 1 of the present application;

[0016] Figure 3 is a schematic diagram of a voice recognition framework according to an embodiment of the present application;

[0017] Figure 4 is a flowchart of a model training method according to embodiment 2 of the present application;

[0018] Figure 5 is a flowchart of a voice recognition method according to embodiment 3 of the present application;

[0019] Figure 6 is a flowchart of a voice recognition method according to embodiment 4 of the present application;

[0020] Figure 7 is a schematic diagram of a speech recognition device according to an embodiment of the application;

[0021] Figure 8 is a schematic diagram of a model training device according to an embodiment of the application;

[0022] Figure 9 is a schematic diagram of a speech recognition device according to an embodiment of the application;

[0023] Figure 10 is a schematic diagram of a speech recognition device according to an embodiment of the application;

[0024] Figure 11 is a structural block diagram of a computer terminal according to an embodiment of the application. DETAILED DESCRIPTION

[0025] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to the process, method, product, or device.

[0027] Traditional multilingual speech recognition needs to rely on a large number of different scenarios and different languages of speech data, and only then can the recognition accuracy rate that customers can use be obtained in general or in some specific fields. However, at present, in addition to English, the multilingual languages are generally low-resource languages, and the traditional speech recognition technology has obvious difficulties in low-resource languages.

[0028] First, multi-language labeling is costly, with poor quality and a long construction period. The model effect is strongly dependent on the labeling amount of speech audio and its corresponding text data. To improve the model recognition rate, a large amount of data needs to be labeled. Currently, the cost of multi-language labeling is more than 3 times that of Chinese data labeling. At the same time, due to the uneven education level of labeling personnel in different languages, the labeling speed is half that of Chinese, and the labeling quality is far lower than that of Chinese. If a multi-language speech recognition model is constructed using traditional methods and reaches a usable level for customers, it will inevitably encounter problems such as high construction cost and long project cycle.

[0029] Second, the generalization ability of speech recognition for low-resource languages is still a challenging problem, with poor recognition rate on the open set. Low-resource languages generally only have reading data, and are extremely lacking in high-value scenario (such as instant messaging, conference, live broadcast, speech, customer service, etc.) data sources. The model is prone to performance degradation due to mismatch between training data and application scenario data, and needs to solve the unsupervised self-learning ability of the model to better adapt to the environment. The current model still has poor transferability between different scenarios and different languages.

[0030] In this application, the sample importance perception adaptive learning method and multi-view consistency regularization method improve the effectiveness in low-resource speech recognition tasks.

[0031] Finally, traditional multi-language speech recognition models different languages separately. The acoustic features such as channel environment, phonemes, and noise in different language training audio can be reused in different languages in theory, but this part of information has not been fully utilized. The above problems need more research and development investment.

[0032] Embodiment 1

[0033] According to the embodiments of the present application, a speech recognition method embodiment is also provided. It should be noted that the steps shown in the flowchart of the drawing can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0034] The method embodiment provided by the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar operation device. Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a speech recognition method according to an embodiment of the present application. As shown in Figure 1As shown, the computer terminal 10 (or mobile device) can include one or more processors 102 (the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can include more or fewer components than those shown in Figure 1 or have a different configuration than that shown in Figure 1 .

[0035] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to herein as "data processing circuits" in general. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements incorporated into the computer terminal 10 (or mobile device) in whole or in part. As referred to in the embodiments of the present application, the data processing circuit serves as a processor to control (for example, selection of a variable resistance terminal path connected to an interface).

[0036] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the voice recognition method in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the voice recognition method described above. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 can further include a memory remotely disposed with respect to the processor 102, which can be connected to the computer terminal 10 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] The transmission device 106 is configured to receive or send data via a network. The network can include, for example, a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module that is configured to communicate with the Internet wirelessly.

[0038] The display can be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0039] It is noted that, in some alternative embodiments, the above-mentioned Figure 1 The computer device (or mobile device) can include hardware elements (including circuitry), software elements (including computer code stored on a computer readable medium), or a combination of both hardware and software elements. It should be noted that Figure 1 is merely one example of a particular implementation and is intended to illustrate the types of components that can be present in the above-mentioned computer device (or mobile device).

[0040] In the above-mentioned operating environment, the present application provides a voice recognition method as shown in the accompanying drawings. Figure 2 Figure 2 is a flowchart of a voice recognition method according to an embodiment of the present application. The method includes the following steps:

[0041] In step S202, voice data obtained by voice collection of a first object is acquired.

[0042] The voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types.

[0043] The plurality of preset language types can include, but are not limited to, Chinese, English, German, Dutch, Spanish, French, Portuguese, Italian, and Polish. The preset language types can be set according to requirements.

[0044] The first object can be an object whose output voice data needs to be recognized, and the voice content of the target language type contained in the output voice data of the first object can be recognized to obtain text information corresponding to the voice content. The first object can be a specific object or a randomly selected object.

[0045] ​In an optional embodiment, the voice content output by the first object can be collected by a voice collection device to obtain the voice data.

[0046] In step S204, the voice data is subjected to feature extraction by using a voice recognition model to obtain acoustic features of the voice data.

[0047] The voice recognition model can be an automatic speech recognition model (ASR).

[0048] In an optional embodiment, the voice data can be subjected to feature extraction by a convolutional subsampling feature encoder in the automatic speech recognition model to obtain acoustic features of the voice data.

[0049] In step S206, the acoustic features are input into the voice recognition model based on the global information and the local information of the acoustic features to perform voice recognition and obtain a target recognition result of the voice content.

[0050] The voice recognition model is obtained by semi-supervised and self-supervised learning of first voice samples and second voice samples, the first voice samples include voice samples of multiple preset language types and do not contain labeled information, and the second voice samples include voice samples of a target language type and labeled information.

[0051] The local information can be acoustic features obtained by randomly masking the global information.

[0052] In an optional embodiment, the acoustic features can be input into an encoder (conformer) of the voice recognition model based on the global information and the local information of the acoustic features to perform voice recognition and obtain a target recognition result of the voice content. The number of encoders can not be limited and can be one or multiple. The number of encoders is taken as an example of 3 in the present application.

[0053] It should be noted that each encoder contains the same model parameters.

[0054] In an optional embodiment, the acoustic feature can be input into a first encoder for recognition according to local information of the acoustic feature to obtain a first recognition result, the acoustic feature can be input into a second encoder for recognition according to global information of the acoustic feature to obtain a second recognition result, the acoustic feature can be data-augmented according to the global information of the acoustic feature, and the data-augmented acoustic feature can be input into a third encoder for recognition to obtain a third recognition result, and the target recognition result of the speech content can be obtained according to the first recognition result, the second recognition result and the third recognition result. The first recognition result, the second recognition result and the third recognition result can correspond to different weights respectively, and the corresponding weight in the final target recognition result can be increased or decreased by adjusting the corresponding weight according to the demand.

[0055] In another optional embodiment, when the acoustic feature is data-augmented according to the global information of the acoustic feature, the acoustic feature can be respectively subjected to strong data augmentation and weak data augmentation, so as to increase multiple perspectives and improve the stability of the result. It should be noted that the strong data augmentation can be a large-amplitude data augmentation on the acoustic feature, for example, a large amount of noise is added, and the weak data augmentation can be a small-amplitude data augmentation on the acoustic feature, for example, a small amount of noise is added.

[0056] It should be noted that the hyperparameter value of the strong data augmentation is larger than that of the weak data augmentation, and the larger hyperparameter represents that more information in the input audio feature is masked and the noise interference is larger.

[0057] The above self-supervised learning mainly uses auxiliary tasks to mine self-supervised information from large-scale unsupervised data, and the network is trained by using the constructed supervised information, so that the network can learn valuable representations for downstream tasks.

[0058] The above semi-supervised learning mainly includes two parts, a small part of labeled data and a large part of unlabeled data. The network is trained using the labeled data, the trained network is used to classify the unlabeled data to make pseudo labels, and the data with correct labels is selected according to a selection principle to train the network again.

[0059] The self-supervised learning and the semi-supervised learning can have the following learning objectives:

[0060]

[0061] wherein, l l is the loss of the second speech sample D l containing labeled data, and l u is the loss of the first speech sample Du where L is the loss, a is a non-negative weight coefficient, and 0 is the model parameter.

[0062] It should be noted that the objective function of multi-task learning can also be extended to multiple unsupervised losses.

[0063] The present application demonstrates the framework in the present application on semi-supervised and self-supervised losses, i.e., sample-pair importance-aware contrastive learning loss and multi-perspective consistency regularization loss. In this work, a framework for speech recognition is proposed, and the formula of the multi-task learning loss function is as follows:

[0064]

[0065] where l SICL (D u , 0) can be a sample-pair importance-aware contrastive learning loss (SICL), and l MPCR (D u , 0) can be a multi-perspective consistency regularization loss (MPCR).

[0066] In the framework of the present application, more semi-supervised and self-supervised losses or multi-perspective consistency regularization losses can be further explored. In the present application, a and b can be set to 0.5, but are not limited thereto. The specific weight setting can be set according to the actual demand.

[0067] Figure 3 is a schematic diagram of a speech recognition framework according to an embodiment of the present application, as Figure 3 indicated, a semi-supervised and self-supervised speech recognition framework (Self-and Semi-Supervised Learning framework for automatic speech recognition, referred to as S 3The L-ASR includes semi-supervised and self-supervised losses, which are selected according to the design of the pre-trained model (wav2vec, wav2vec 2.0) that can include a convolutional subsampling feature encoder, a conformer context encoder, and a vector quantizer. The application first uses a self-supervised learning method to obtain speech sample features of unlabeled data, then constructs an importance-aware contrastive loss on the data sample pair according to the speech sample features of the unlabeled data and the speech sample features of the labeled data, and finally, multi-angle consistency regularization can be used to improve the robustness of the model to worse cases.

[0068] As shown in Figure 3 The input label can be enhanced in the application, and then a multi-view consistency regularization loss is used to maintain consistency in the worst case. The features input to the conformer block can be obtained by masking the potential speech representation in the features, while the speech input to the quantization block is not masked. The contrastive loss aware of the importance of the sample pair can be used to learn the robustness of the enhanced speech features of the masked and unmasked speech sample pair. The randomly masked speech sample is sent to the encoder block, and the Connectionist temporal classification (CTC) loss is used for learning to realize the training of the S 3 L-ASR through multi-task learning. It should be noted that when the speech sample is sent to the neural network subsampling layer (CNN subsampling), weak data augmentation and strong data augmentation can be performed on the speech sample respectively to construct the consistency regularization loss from multiple perspectives.

[0069] By the above steps, the voice data obtained by voice collection on the first object is obtained, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; acoustic features of the voice data are obtained by using a voice recognition model to perform feature extraction on the voice data; the acoustic features are input to the voice recognition model based on global information and local information of the acoustic features to perform voice recognition, and a target recognition result of the voice content is obtained, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of the plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information, thereby improving the accuracy of voice data recognition; in this application, the voice recognition model can be trained by using multi-task semi-supervised and self-supervised learning, which can improve the generalization ability of the voice recognition model, and the voice recognition model can also perform high-precision recognition on voice content of a language type with a small number of samples, thereby improving the recognition accuracy of voice data and solving the technical problem of low recognition accuracy of the voice recognition model in related technologies.

[0070] In the above embodiments of the present application, the method further includes: performing multiple data augmentations on the first voice samples to obtain an augmented sample set; performing semi-supervised learning on the voice recognition model using the first voice samples and the augmented sample set to obtain a semi-supervised loss value; performing self-supervised learning on the voice recognition model using the second voice samples to obtain a self-supervised loss value; obtaining a target loss value corresponding to the voice recognition model based on the self-supervised loss value and the semi-supervised loss value; and adjusting model parameters of the voice recognition model based on the target loss value.

[0071] The augmented sample set described above can be a sample set obtained by augmenting the first voice samples.

[0072] In an optional embodiment, the first speech sample can be subjected to weak data enhancement and strong data enhancement respectively to obtain an enhanced sample set, the speech recognition model can be subjected to semi-supervised learning by using the first speech sample and the enhanced sample set, optionally, part of the features of the first speech sample can be shielded to obtain shielded features, and the contrastive learning of sample pair importance perception can be performed according to the first speech sample and the shielded features, so as to determine the perception ability of the speech recognition model to the important area, so that the contrastive learning loss of sample pair importance perception can be obtained, and the multi-view consistency regularization loss can be obtained according to the weak data enhancement and the strong data enhancement in the first speech sample and the enhanced sample set, so as to improve the recognition ability of the speech recognition model to the sample in different views, and the semi-supervised loss value is obtained according to the contrastive learning loss of sample pair importance perception and the multi-view consistency regularization loss, so as to improve the stability, robustness and convergence of the semi-supervised loss value.

[0073] The target language type described above can be a language type with a smaller number of samples in a plurality of preset language types. For some small languages, there may not be many speech samples.

[0074] In another optional embodiment, the speech recognition model can be subjected to self-supervised learning by using the second speech sample, in the process of self-supervised learning, part of the speech features of the second speech sample can be shielded, the shielded speech features can be recognized to obtain a recognition result, and the self-supervised learning of the second speech sample can be completed by using the recognition result and the standard information. By shielding part of the speech features of the second speech sample, the perception ability of the speech recognition model to the area that does not appear can be trained, so as to further improve the speech recognition ability of the speech recognition model.

[0075] In yet another optional embodiment, for the self-supervised loss value and the semi-supervised loss value, corresponding weight values can be set to adjust the proportion of the self-supervised loss value and the semi-supervised loss value, so as to more flexibly adapt to the training process of the speech recognition model. By adjusting the model parameters of the speech recognition model by using the target loss value, the speech recognition model can have stronger speech recognition ability for the language type with fewer samples, so as to improve the recognition accuracy of the speech recognition model.

[0076] In the above embodiments of the present application, the speech recognition model is subjected to semi-supervised learning by using the first speech sample and the enhanced sample set to obtain a semi-supervised loss value, including: the speech recognition model is subjected to semi-supervised learning by using the first speech sample to obtain a first loss value of the speech recognition model; the speech recognition model is subjected to semi-supervised learning by using the first speech sample and the enhanced sample set to obtain a second loss value of the speech recognition model; and a weighted sum of the first loss value and the second loss value is obtained to obtain the semi-supervised loss value.

[0077] The first loss value can be a contrastive loss of sample pair importance perception.

[0078] The second loss value can be a multi-view consistency regularization loss value.

[0079] In an optional embodiment, a part of the features in the sample features of the first speech sample can be masked, and a contrast can be made between the masked features and all the features of the first speech sample, so as to obtain the contrastive loss of sample pair importance perception, so as to improve the perception ability of the language recognition model to the important regions in the speech samples of the plurality of preset language types; the view of the sample can be increased by enhancing the sample set, and the multi-view consistency regularization loss value can be obtained by recognizing the first speech sample and the enhanced sample set, so as to improve the speech recognition ability of the speech recognition model in multiple recognitions.

[0080] In another optional embodiment, for the first loss value and the second loss value, corresponding weight values can be set to adjust the proportion of the first loss value and the second loss value, so as to more flexibly adapt to the training process of the speech recognition model, and the generalization ability of the speech recognition model can be improved by adjusting the model parameters of the speech recognition model through the semi-supervised loss value.

[0081] In the above embodiments of the application, the speech recognition model is subjected to semi-supervised learning by using the first speech sample, and the first loss value of the speech recognition model includes: extracting features of the first speech sample to obtain first sample features of the first speech sample; masking a part of the features in the first sample features to obtain first mask features; quantizing the first sample features to obtain quantized features; and obtaining the first loss value based on the first mask features and the quantized features.

[0082] In an optional embodiment, the first speech sample can be subjected to feature extraction by using a CNN subsampling, the first sample features of the first speech sample can be obtained, a part of the features with labels in the first sample features can be masked to obtain the first mask features, the first sample features can be quantized by using a quantization block, so as to represent the floating-point numbers in the first sample features by fixed-point numbers to obtain the quantized features, and the first loss value can be obtained by comparing the first mask features and the quantized features, if the difference between the two is small, it indicates that the speech recognition model has strong global recognition ability, and the influence of details can be ignored.

[0083] In the above embodiments of this application, obtaining the first loss value based on the first mask feature and the quantization feature includes: determining the positive sample feature and the negative sample feature corresponding to the first mask feature in the quantization feature, wherein the positive sample feature and the first mask feature correspond to the same first speech sample, and the negative sample feature and the first mask feature correspond to different first speech samples; obtaining the similarity between the first mask feature and the positive sample feature and the negative sample feature respectively, to obtain the first similarity and the second similarity; obtaining the sample pair importance factor of the first mask feature and the negative sample feature; and obtaining the first loss value based on the first similarity, the second similarity, and the sample pair importance factor.

[0084] The aforementioned sample pair importance factor refers to the features in the first mask feature that are related to the negative sample feature. Optionally, the context features of the masked feature can be used as the sample pair importance factor of the negative sample feature.

[0085] The first similarity mentioned above can be the maximum similarity between the first mask feature and the positive sample feature.

[0086] The second similarity mentioned above can be the minimum similarity between the first mask feature and the negative sample feature.

[0087] In one optional embodiment, positive sample features belonging to the first mask feature and negative sample features not belonging to the first mask feature can be determined in the quantized features. A first similarity between the first mask feature and the positive sample features can be obtained, and a second similarity between the first mask feature and the negative sample features can be obtained. The importance factor of the sample pair between the first mask feature and the negative sample features can be obtained. A first loss value can be obtained based on the first similarity, the second similarity, and the sample pair importance factor. The parameters of the speech recognition model can be adjusted using the first loss value, which can better represent the masked features, thereby improving the speech recognition model's ability to recognize important regions.

[0088] Assume there are N pairs of speech samples and Where x(i j ) and x(i k The samples are obtained by processing x(i) in different ways, and these samples can be sent to S. 3 L, and output z(i) respectively j ) and z(i k This can be achieved by maximizing the distance between different samples x(m) and x(n). By using adaptive learning (CL) to find feature representations that meet certain conditions, after processing the same sample data i, x(i) can be obtained based on the processing. j ) and x(i k Minimize distance

[0089]

[0090] However, the traditional CL considers different samples equally important, for which the present application proposes a sample pair importance perception contrast loss, which can introduce a pair of importance factors γ to help learn a better representation, where the pair of importance factors γ t is introduced to make the model pay more attention to difficult-to-separate sample pairs rather than already separated sample pairs. Given the context output c t , the model needs to identify the true quantized latent speech representation among a set of k+1 quantization candidates Q t , where K can be uniformly sampled from other steps in the unified expectation, and the loss can be defined as:

[0091]

[0092] where the cosine similarity between quantizes the latent speech representation and the context representation.

[0093] In the above embodiments of the present application, the first speech sample and the enhanced sample set are used for semi-supervised learning of the speech recognition model to obtain a second loss value of the speech recognition model, which includes: performing feature extraction on the first speech sample and the enhanced sample set respectively to obtain first sample features of the first speech sample and an enhanced sample feature set of the enhanced sample set; using the speech recognition model to perform speech recognition on the first sample features and the enhanced sample feature set respectively to obtain a first sample recognition result of the first speech sample and an enhanced sample recognition result set of the enhanced sample set; and obtaining the second loss value based on the first sample recognition result and the enhanced sample recognition result set.

[0094] The enhanced sample set described above can include an enhanced sample that enhances the first speech sample once, or an enhanced sample that enhances the first speech sample multiple times, for example, a sample that weakly enhances and strongly enhances the first speech sample respectively; but is not limited thereto, and the enhanced sample set after enhancing the first speech sample can be set according to actual conditions.

[0095] In an optional embodiment, by identifying the first sample feature, a first sample identification result of the first speech sample can be obtained, by identifying the enhanced sample feature set, an enhanced sample identification result set of the enhanced sample feature set can be obtained, by comparing the first sample identification result and the enhanced sample identification result set, the speech recognition ability of the speech recognition model can be determined. The recognition ability of the same speech sample is enhanced and not enhanced. If the second loss value is larger, it means that the speech recognition ability of the speech recognition model is easy to be disturbed. The second loss value can improve the robustness of the speech recognition model, thereby improving the recognition accuracy of the speech recognition model for speech data.

[0096] At present, K times enhancement is generally performed on each unlabeled sample, the maximum consistency loss is found for each unlabeled sample, the selected consistency loss and the CTC loss of the labeled sample with high confidence can be minimized, in the traditional consistency regularization method, only one or two enhanced data is considered, and in the present application, an improved multi-view consistency regularization method is proposed, which can minimize the maximum inconsistency distance between the original unlabeled sample and multiple enhanced views. The formula of the multi-view consistency regularization loss is as follows:

[0097]

[0098] wherein, is the jth enhanced data, and the same data can be expanded to obtain the ith unlabeled sample. For example, the frequency can be enhanced or the noise can be increased.

[0099] In the above embodiments of the present application, based on the first sample identification result and the enhanced sample identification result set, the second loss value is obtained, including: using the cross-entropy loss to process the first sample identification result and each enhanced sample identification result in the enhanced sample identification result set to obtain a set of original loss values corresponding to the enhanced sample set; and obtaining the second loss value based on the maximum loss value in the set of original loss values.

[0100] In an optional embodiment, when the enhanced sample identification result set includes multiple enhanced sample identification results, the first sample identification result and the original loss value of each enhanced sample identification result can be obtained, thereby obtaining a set of original loss values corresponding to the enhanced sample set. By sorting the multiple original loss values in the set of original loss values, the maximum loss value in the set of original loss values can be obtained, and the maximum loss value can be determined as the second loss value. By training the speech recognition model based on the maximum loss value, the speech recognition model can converge faster, thereby improving the robustness of the speech recognition model.

[0101] Optionally, the second loss value can be obtained by randomly selecting from a plurality of original loss values in the original loss set. The specific manner of determining the second loss value can also be flexibly set according to requirements.

[0102] In the above embodiments of the present application, the self-supervised learning of the speech recognition model by using the second speech sample to obtain the self-supervised loss value includes: performing feature extraction on the second speech sample to obtain second sample features; performing masking on part of the second sample features to obtain second mask features; performing speech recognition on the second mask features by using the speech recognition model to obtain a second sample recognition result of the second speech sample; and obtaining the self-supervised loss value based on the second sample recognition result and the annotation information.

[0103] In an optional embodiment, the second sample features can be obtained by performing feature extraction on the second speech sample, the second mask features can be obtained by performing masking on part of the second sample features, the second sample recognition result of the second speech sample can be obtained by performing speech recognition on the second mask features by using the speech recognition model, the recognition ability of the speech recognition model for the hidden area can be trained by masking part of the features, so that the speech recognition model can make predictions based on context information, the self-supervised loss value can be obtained by using the second sample recognition result and the annotation information, and the prediction ability of the speech recognition model for the hidden features can be improved by using the self-supervised loss value.

[0104] In the present application, a self-supervised learning and semi-supervised learning framework for automatically performing speech recognition is proposed, which has a sample pair importance-aware contrastive loss and multi-view consistency regularization. In recent years, self-supervised learning (SSL) has attracted more and more attention, as it can use large amounts of unlabeled data and labeled data to avoid a large amount of annotation work by humans. Contrastive learning has achieved many successful applications in speech processing, however, existing research ignores the unequal importance between sample pairs, therefore, the present application introduces equal importance in the contrastive loss in order to improve the contrastiveness and robustness of the learned speech representation, in order to improve the generalization ability of self-supervised learning, the present application adds a worst-case consistency regularization loss to minimize the maximum inconsistency between the original unlabeled data and the connected multiple enhanced views, then, the present application performs multi-task semi-supervised and self-supervised learning on a multi-language speech recognition task, the proposed method is evaluated on existing data sets, and the implementation shows that the method proposed in the present application can have comparable performance to the baseline method in a low-resource multi-language speech recognition task.

[0105] The background of the present application scheme implementation is as follows:

[0106] Currently, automatic speech recognition (ASR) has achieved remarkable success, which can be attributed to two complementary aspects: 1) more effective and larger deep neural networks designed for automatic speech recognition; 2) training on a large amount of data. However, in practice, unlike common languages such as English and Mandarin, which have sufficient training data, due to the scarcity of audio and the huge labor resources consumed in the transcription process, the vast majority of spoken languages in the world have only low-resource data, and in this case, it is impractical to use traditional methods for self-supervised learning of low-resource languages. In recent years, self-supervised learning has attracted increasing research attention, which can learn from large amounts of unlabeled data and labeled data, and can avoid the use of a large number of artificial labels. With the support of self-supervised learning, semi-supervised learning has great potential in solving the problem of low-resource multi-speech recognition.

[0107] Most advanced semi-supervised learning frameworks are based on the idea of pursuing consistent model predictions on input noise on unlabelled data, which is called consistent regularization, and another search paradigm for semi-supervised learning is iterative self-training, which uses data augmentation (SpecAugment) for noise student training on a speech recognition model (Conformer) and pre-training through wav2vec or wav2vec 2.0. On this basis, the present application proposes a multi-view consistency regularization training method, which minimizes the maximum inconsistency loss on the uncertainty set generated by data augmentation, and self-supervised speech representation learning aims to learn useful speech representations without relying on artificial labels, which can solve the learning difficulty in the low-resource field. A mainstream method is semi-supervised pre-training or representation learning, which has received increasing attention in speech recognition. wav2vec and wav2vec 2.0 apply semi-supervised contrastive pre-training and show good results, and Unispeech proposes a unified speech representation learning that can use supervised CTC loss and speech-aware contrastive loss through multi-task learning, and learn from labeled and unlabeled data

[0108] Unispeech improves the ability to learn speaker statements by introducing corpus mixture augmentation and corpus contrastive loss. The speech pre-training model (WavLM) learns the prediction and denoising of the masked speech in the pre-training process using the mask prediction loss and the expected mixture data augmentation method.

[0109] In the traditional contrastive learning method, the importance of the sample pair is considered to be equal, but in the learning process, some sample pairs are difficult to learn, and some are easy to learn, and the application proposes a sample pair importance aware contrastive learning (SICL) loss to improve the consistency regularization. The traditional consistency regularization method considers the equal importance between different sample pairs, however, in the training process, some sample pairs are difficult to train their consistency, and others are easy to train, therefore, the application introduces multi-view consistency to improve the effectiveness of consistency regularization and as much as possible to choose the poor case for learning.

[0110] In semi-supervised learning, self-supervised contrastive loss and consistency regularization are two auxiliary losses, in this work, the application proposes a new semi-supervised and self-supervised framework for automatic speech recognition, the contributions of the application are as follows:

[0111] 1) The application proposes a sample pair importance aware adaptive learning method to improve the robustness of semi-supervised speech recognition on defect-free data.

[0112] 2) The application proposes a multi-view consistency regularization loss that can improve the training stability and convergence of semi-supervised automatic recognition.

[0113] 3) The method in the application shows the effectiveness of the multi-language speech recognition task on the industrial dataset, which can improve the recognition accuracy of speech recognition.

[0114] It should be noted that for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the application is not limited by the action order described, because according to the application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the application.

[0115] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, or network device, etc.) execute the method of each embodiment of the application.

[0116] Embodiment 2

[0117] According to the embodiments of the present application, a model training method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from here.

[0118] Figure 4 is a flowchart of a model training method according to Embodiment 2 of the present application, as shown in Figure 4 The method can include the following steps:

[0119] Step S402, obtaining a first speech sample and a second speech sample.

[0120] Among them, the first speech sample contains speech samples of multiple preset language types, and does not contain annotation information, and the second speech sample contains language samples of a target language type and annotation information, and the target language type is any one of the multiple preset language types.

[0121] Step S404, using the first speech sample and the second speech sample to perform semi-supervised and self-supervised learning on the initial recognition model to obtain a speech recognition model.

[0122] Among them, the speech recognition model is used to perform speech recognition on the speech content contained in the speech data based on the global information and the local information of the speech data to obtain a target recognition result of the speech content, and the language type corresponding to the speech content is the target language type.

[0123] By the above steps, the first speech sample and the second speech sample are obtained, wherein the first speech sample contains speech samples of multiple preset language types and does not contain annotation information, and the second speech sample contains a language sample of a target language type and annotation information, the target language type being any one of the multiple preset language types, the initial recognition model is subjected to semi-supervised and self-supervised learning by using the first speech sample and the second speech sample, and a speech recognition model is obtained; wherein the speech recognition model is used to perform speech recognition on speech content contained in speech data based on global information and local information of the speech data to obtain a target recognition result of the speech content, and the language type corresponding to the speech content is the target language type, thereby improving the accuracy of speech data recognition; in the present application, the speech recognition model can be trained by using multi-task semi-supervised and self-supervised learning, the generalization ability of the speech recognition model can be improved, the speech recognition model can also perform high-precision recognition on speech content of a language type with a small number of samples, thereby improving the recognition accuracy of speech data, and further solving the technical problem of low recognition accuracy of the speech recognition model in the related art.

[0124] In the above embodiments of the present application, the initial recognition model is subjected to semi-supervised and self-supervised learning by using the first speech sample and the second speech sample to obtain the speech recognition model, which includes: performing multiple data augmentations on the first speech sample to obtain an augmented sample set; performing semi-supervised learning on the initial recognition model by using the first speech sample and the augmented sample set to obtain a semi-supervised loss value; performing self-supervised learning on the initial recognition model by using the second speech sample to obtain a self-supervised loss value; obtaining a target loss value corresponding to the initial recognition model based on the self-supervised loss value and the semi-supervised loss value; and adjusting model parameters of the initial recognition model based on the target loss value to obtain the speech recognition model.

[0125] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0126] Embodiment 3

[0127] According to the embodiments of the present application, a speech recognition method is also provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0128] Figure 5 is a flowchart of a speech recognition method according to Embodiment 3 of the present application, as shown in Figure 5 The method can include the following steps:

[0129] In step S502, in response to an input instruction acting on the operation interface, a voice collection device is driven to collect voice of the first object to obtain voice data.

[0130] The voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types.

[0131] In step S504, in response to a voice recognition instruction acting on the operation interface, a target recognition result of the voice content is displayed on the operation interface.

[0132] The target recognition result is obtained by inputting acoustic features of the voice data into a voice recognition model for voice recognition based on global information and local information of the acoustic features, the acoustic features are obtained by performing feature extraction on the voice data using the voice recognition model, the voice recognition model is obtained by performing semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information.

[0133] Through the above steps, in response to an input instruction acting on the operation interface, a voice collection device is driven to collect voice of the first object to obtain voice data. The voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types. In response to a voice recognition instruction acting on the operation interface, a target recognition result of the voice content is displayed on the operation interface. The target recognition result is obtained by inputting acoustic features of the voice data into a voice recognition model for voice recognition based on global information and local information of the acoustic features. The acoustic features are obtained by performing feature extraction on the voice data using the voice recognition model. The voice recognition model is obtained by performing semi-supervised and self-supervised learning using first voice samples and second voice samples. The first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information. The second voice samples contain language samples of the target language type and labeled information. This improves the accuracy of voice data recognition. In this application, the voice recognition model can be trained using multi-task semi-supervised and self-supervised learning, which improves the generalization ability of the voice recognition model. For voice content of a language type with a small number of samples, the voice recognition model can also perform high-precision recognition, thereby improving the accuracy of voice data recognition and solving the technical problem of low recognition accuracy of the voice recognition model in related technologies.

[0134] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0135] Embodiment 4

[0136] According to the embodiments of the present application, a speech recognition method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0137] Figure 6 is a flowchart of a speech recognition method according to Embodiment 4 of the present application, as shown in Figure 6 The method can include the following steps:

[0138] Step S602, obtaining speech data obtained by speech collection on a first object by calling a first interface.

[0139] The first interface includes a first parameter, and the parameter value of the first parameter is the speech data, which contains speech content of a target language type, and the target language type is any one of a plurality of preset language types.

[0140] The first interface described above can be an interface for data exchange between a server and a client. The client can pass the speech data into the interface function as the first parameter of the interface function, so as to achieve the purpose of uploading the speech data to the cloud server.

[0141] Step S604, performing feature extraction on the speech data by using a speech recognition model to obtain acoustic features of the speech data.

[0142] Step S606, inputting the acoustic features into the speech recognition model based on global information and local information of the acoustic features to perform speech recognition and obtain a target recognition result of the speech content.

[0143] The speech recognition model is obtained by semi-supervised and self-supervised learning using a first speech sample and a second speech sample. The first speech sample contains speech samples of a plurality of preset language types and does not contain annotation information. The second speech sample contains language samples of a target language type and annotation information.

[0144] Step S608, outputting the target recognition result by calling a second interface.

[0145] The second interface includes a second parameter, and the parameter value of the second parameter is the target recognition result.

[0146] The second interface in the above steps can be an interface for data exchange between a cloud server and a client. The cloud server can pass the target recognition result into the interface function as the second parameter of the interface function, so as to achieve the purpose of issuing the target recognition result to the client.

[0147] By the above steps, the voice data obtained by voice collection on the first object is acquired by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter is the voice data, the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; acoustic features of the voice data are obtained by performing feature extraction on the voice data by using a voice recognition model; the acoustic features are input to the voice recognition model for voice recognition based on global information and local information of the acoustic features, to obtain a target recognition result of the voice content, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning of first voice samples and second voice samples, the first voice samples contain voice samples of the plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information; and the target recognition result is output by calling a second interface, wherein the second interface includes a second parameter, a parameter value of the second parameter is the target recognition result, and the accuracy of voice data recognition is improved. In the present application, the voice recognition model can be trained by multi-task semi-supervised and self-supervised learning, the generalization ability of the voice recognition model can be improved, the voice recognition model can also perform high-precision recognition on voice content of a language type with a small number of samples, thereby improving the recognition accuracy of voice data, and further solving the technical problem of low recognition accuracy of the voice recognition model in the related art.

[0148] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0149] Embodiment 5

[0150] According to the embodiments of the present application, a voice recognition device for implementing the voice recognition method is further provided, Figure 7 is a schematic diagram of a voice recognition device according to Embodiment 5 of the present application, as Figure 7 shown, the device 700 includes an acquisition module 702, a feature extraction module 704, and an input module 706.

[0151] The acquisition module is configured to acquire voice data obtained by voice collection on the first object, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; the feature extraction module is configured to perform feature extraction on the voice data by using a speech recognition model to obtain acoustic features of the voice data; and the input module is configured to input the acoustic features to the speech recognition model based on global information and local information of the acoustic features to perform speech recognition and obtain a target recognition result of the voice content, wherein the speech recognition model is obtained by semi-supervised and self-supervised learning based on first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information.

[0152] It should be noted that the acquisition module 702, the feature extraction module 704, and the input module 706 correspond to steps S202 to S206 in Embodiment 1, and the three modules have the same instances and application scenarios as those in the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the modules as part of the device can run in the computer terminal provided in Embodiment 1.

[0153] In the above embodiments of the present application, the device further includes a data enhancement module, a semi-supervised learning module, a self-supervised learning module, and an adjustment module.

[0154] The data enhancement module is configured to perform multiple data enhancements on the first voice samples to obtain an enhanced sample set; the semi-supervised learning module is configured to perform semi-supervised learning on the speech recognition model by using the first voice samples and the enhanced sample set to obtain a semi-supervised loss value; the self-supervised learning module is configured to perform self-supervised learning on the speech recognition model by using the second voice samples to obtain a self-supervised loss value; the semi-supervised learning module is configured to obtain a target loss value corresponding to the speech recognition model based on the self-supervised loss value and the semi-supervised loss value; and the adjustment module is configured to adjust model parameters of the speech recognition model based on the target loss value.

[0155] In the above embodiments of the present application, the semi-supervised learning module is further configured to perform semi-supervised learning on the speech recognition model by using the first voice samples to obtain a first loss value of the speech recognition model; the semi-supervised learning module is further configured to perform semi-supervised learning on the speech recognition model by using the first voice samples and the enhanced sample set to obtain a second loss value of the speech recognition model; and the semi-supervised learning module is further configured to obtain a weighted sum of the first loss value and the second loss value to obtain the semi-supervised loss value.

[0156] In the above embodiments of the present application, the semi-supervised learning module is further configured to perform feature extraction on the first speech sample to obtain first sample features of the first speech sample; the semi-supervised learning module is further configured to mask part of the first sample features to obtain first mask features; the semi-supervised learning module is further configured to quantize the first sample features to obtain quantized features; and the semi-supervised learning module is further configured to obtain the first loss value based on the first mask features and the quantized features.

[0157] In the above embodiments of the present application, the semi-supervised learning module is further configured to determine positive sample features and negative sample features corresponding to the first mask features in the quantized features, wherein the positive sample features and the first mask features correspond to the same first speech sample, and the negative sample features and the first mask features correspond to different first speech samples; the semi-supervised learning module is further configured to obtain similarities of the first mask features with the positive sample features and the negative sample features respectively to obtain first similarity and second similarity; the semi-supervised learning module is further configured to obtain a sample pair importance factor of the first mask features and the negative sample features; and the semi-supervised learning module is further configured to obtain the first loss value based on the first similarity, the second similarity and the sample pair importance factor.

[0158] In the above embodiments of the present application, the semi-supervised learning module is further configured to perform feature extraction on the first speech sample and the enhanced sample set respectively to obtain first sample features of the first speech sample and enhanced sample feature sets of the enhanced sample set; the semi-supervised learning module is further configured to perform speech recognition on the first sample features and the enhanced sample feature sets respectively by using the speech recognition model to obtain a first sample recognition result of the first speech sample and an enhanced sample recognition result set of the enhanced sample set; and the semi-supervised learning module is further configured to obtain the second loss value based on the first sample recognition result and the enhanced sample recognition result set.

[0159] In the above embodiments of the present application, the semi-supervised learning module is further configured to process each enhanced sample recognition result in the first sample recognition result and the enhanced sample recognition result set by using a cross-entropy loss to obtain an original loss value set corresponding to the enhanced sample set; and the semi-supervised learning module is further configured to obtain the second loss value based on a maximum loss value in the original loss value set.

[0160] In the above embodiments of the present application, the self-supervised learning module is configured to perform feature extraction on the second speech sample to obtain second sample features; the self-supervised learning module is further configured to mask part of the second sample features to obtain second mask features; the self-supervised learning module is further configured to perform speech recognition on the second mask features by using the speech recognition model to obtain a second sample recognition result of the second speech sample; and the self-supervised learning module is further configured to obtain a self-supervised loss value based on the second sample recognition result and the labeled information.

[0161] It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0162] Embodiment 6

[0163] According to the embodiments of the present application, a model training device for implementing the above model training method is further provided, Figure 8 is a schematic diagram of a model training device according to Embodiment 6 of the present application, as Figure 8 shown, the device 800 includes an acquisition module 802, a supervised learning module 804.

[0164] The acquisition module is configured to acquire first speech samples and second speech samples, wherein the first speech samples contain speech samples of multiple preset language types and do not contain labeled information, and the second speech samples contain language samples of a target language type and labeled information, the target language type being any one of the multiple preset language types; the supervised learning module is configured to perform semi-supervised and self-supervised learning on the initial recognition model using the first speech samples and the second speech samples to obtain a speech recognition model, wherein the speech recognition model is configured to perform speech recognition on speech content contained in speech data based on global information and local information of the speech data to obtain a target recognition result of the speech content, and the language type corresponding to the speech content being the target language type.

[0165] It should be noted that the acquisition module 802 and the supervised learning module 804 correspond to steps S402 to S404 in Embodiment 2, and the instances and application scenarios realized by the two modules in the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal provided in Embodiment 1.

[0166] In the above embodiments of the present application, the supervised learning module is further configured to perform multiple data augmentations on the first speech samples to obtain an augmented sample set; the supervised learning module is further configured to perform semi-supervised learning on the initial recognition model using the first speech samples and the augmented sample set to obtain a semi-supervised loss value; the supervised learning module is further configured to perform self-supervised learning on the initial recognition model using the second speech samples to obtain a self-supervised loss value; the supervised learning module is further configured to obtain a target loss value corresponding to the initial recognition model based on the self-supervised loss value and the semi-supervised loss value; and the supervised learning module is further configured to adjust model parameters of the initial recognition model based on the target loss value to obtain the speech recognition model.

[0167] It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0168] Embodiment 7

[0169] According to the embodiments of the present application, a voice recognition device for implementing the voice recognition method is further provided, Figure 9 is a schematic diagram of a voice recognition device according to Embodiment 7 of the present application, as Figure 9 shown, the device 900 includes a driving module 902, a display module 904.

[0170] The driving module is configured to drive the voice collection device to collect voice of the first object to obtain voice data in response to an input instruction acting on the operation interface, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; the display module is configured to display a target recognition result of the voice content on the operation interface in response to a voice recognition instruction acting on the operation interface, wherein the target recognition result is obtained by inputting acoustic features based on global information and local information of the voice data into a voice recognition model for voice recognition, the acoustic features are obtained by performing feature extraction on the voice data using the voice recognition model, and the voice recognition model is obtained by performing semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information.

[0171] It should be noted that the driving module 902 and the display module 904 correspond to steps S502 to S504 in Embodiment 3, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the device can run in the computer terminal provided in Embodiment 1.

[0172] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same scheme, application scenario and implementation process as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0173] Embodiment 8

[0174] According to the embodiments of the present application, a voice recognition device for implementing the voice recognition method is further provided, Figure 10 is a schematic diagram of a voice recognition device according to Embodiment 8 of the present application, as Figure 10 shown, the device 1000 includes a collection module 1002, a feature extraction module 1004, an input module 1006, and an output module 1008.

[0175] The collection module is configured to obtain voice data obtained by voice collection on the first object by calling a first interface, the first interface includes a first parameter, a parameter value of the first parameter is the voice data, the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types. The feature extraction module is configured to extract acoustic features of the voice data by using a speech recognition model. The input module is configured to input the acoustic features to the speech recognition model for speech recognition based on global information and local information of the acoustic features, to obtain a target recognition result of the voice content. The speech recognition model is obtained by semi-supervised and self-supervised learning of first voice samples and second voice samples, the first voice samples include voice samples of a plurality of preset language types and do not include labeled information, and the second voice samples include language samples of the target language type and labeled information. The output module is configured to output the target recognition result by calling a second interface, the second interface includes a second parameter, and a parameter value of the second parameter is the target recognition result.

[0176] It should be noted that the collection module 1002, the feature extraction module 1004, the input module 1006, and the output module 1008 correspond to steps S602 to S608 in Embodiment 4, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the solutions disclosed in Embodiment 1. It should be noted that the above modules as part of the device can run in the computer terminal provided in Embodiment 1.

[0177] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios, implementation processes, and solutions as those provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

[0178] Embodiment 9

[0179] The embodiments of the present application can provide a computer terminal, which can be any one of the computer terminal devices in the computer terminal group. Alternatively, in the present embodiment, the above computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0180] Alternatively, in the present embodiment, the above computer terminal can be located in at least one network device of a plurality of network devices of a computer network.

[0181] In the embodiment, the computer terminal can execute program codes of the following steps in the speech recognition method: obtaining speech data obtained by performing speech collection on a first object, wherein the speech data contains speech content of a target language type, and the target language type is any one of a plurality of preset language types; performing feature extraction on the speech data by using a speech recognition model to obtain acoustic features of the speech data; inputting the acoustic features into the speech recognition model based on global information and local information of the acoustic features to perform speech recognition, and obtaining a target recognition result of the speech content, wherein the speech recognition model is obtained by performing semi-supervised and self-supervised learning on a first speech sample and a second speech sample, the first speech sample contains speech samples of a plurality of preset language types and does not contain labeled information, and the second speech sample contains language samples of the target language type and labeled information.

[0182] Optionally, Figure 11 is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in the figure, the computer terminal A can include one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency module, an audio module, and a display. Figure 11

[0183] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the speech recognition method and device in the embodiments of the present application. The processor performs various function applications and data processing by running the software programs and modules stored in the memory, that is, implements the speech recognition method described above. The memory can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged relative to the processor, and the remote memory can be connected to the terminal A through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0184] ​The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining voice data obtained by voice collection on a first object, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; performing feature extraction on the voice data using a speech recognition model to obtain acoustic features of the voice data; inputting the acoustic features into the speech recognition model based on global information and local information of the acoustic features to perform speech recognition and obtain a target recognition result of the voice content, wherein the speech recognition model is obtained by semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information.

[0185] Optionally, the processor can further execute program codes of the following steps: performing multiple data augmentations on the first voice samples to obtain an augmented sample set; performing semi-supervised learning on the speech recognition model using the first voice samples and the augmented sample set to obtain a semi-supervised loss value; performing self-supervised learning on the speech recognition model using the second voice samples to obtain a self-supervised loss value; obtaining a target loss value corresponding to the speech recognition model based on the self-supervised loss value and the semi-supervised loss value; and adjusting model parameters of the speech recognition model based on the target loss value.

[0186] Optionally, the processor can further execute program codes of the following steps: performing semi-supervised learning on the speech recognition model using the first voice samples to obtain a first loss value of the speech recognition model; performing semi-supervised learning on the speech recognition model using the first voice samples and the augmented sample set to obtain a second loss value of the speech recognition model; obtaining a weighted sum of the first loss value and the second loss value to obtain the semi-supervised loss value.

[0187] Optionally, the processor can further execute program codes of the following steps: performing feature extraction on the first voice samples to obtain first sample features of the first voice samples; performing masking on part of the first sample features to obtain first mask features; performing quantization on the first sample features to obtain quantized features; and obtaining the first loss value based on the first mask features and the quantized features.

[0188] Optionally, the processor can further execute program codes of the following steps: determining positive sample features and negative sample features corresponding to the first mask feature in the quantified features, wherein the positive sample features and the first mask feature correspond to the same first speech sample, and the negative sample features and the first mask feature correspond to different first speech samples; obtaining similarities of the first mask feature with the positive sample features and the negative sample features respectively, to obtain a first similarity and a second similarity; obtaining a sample pair importance factor of the first mask feature and the negative sample features; and obtaining the first loss value based on the first similarity, the second similarity and the sample pair importance factor.

[0189] Optionally, the processor can further execute program codes of the following steps: performing feature extraction on the first speech sample and the enhanced sample set respectively to obtain first sample features of the first speech sample and an enhanced sample feature set of the enhanced sample set; performing speech recognition on the first sample features and the enhanced sample feature set respectively by using the speech recognition model to obtain a first sample recognition result of the first speech sample and an enhanced sample recognition result set of the enhanced sample set; and obtaining the second loss value based on the first sample recognition result and the enhanced sample recognition result set.

[0190] Optionally, the processor can further execute program codes of the following steps: processing each enhanced sample recognition result in the first sample recognition result and the enhanced sample recognition result set by using a cross-entropy loss to obtain an original loss value set corresponding to the enhanced sample set; and obtaining the second loss value based on a maximum loss value in the original loss value set.

[0191] Optionally, the processor can further execute program codes of the following steps: performing feature extraction on the second speech sample to obtain second sample features; performing masking on part of the second sample features to obtain second mask features; performing speech recognition on the second mask features by using the speech recognition model to obtain a second sample recognition result of the second speech sample; and obtaining the self-supervised loss value based on the second sample recognition result and the labeled information.

[0192] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: obtaining first speech samples and second speech samples, wherein the first speech samples contain speech samples of multiple preset language types and do not contain labeled information, and the second speech samples contain language samples of a target language type and labeled information, and the target language type is any one of the multiple preset language types; and performing semi-supervised and self-supervised learning on an initial recognition model by using the first speech samples and the second speech samples to obtain a speech recognition model, wherein the speech recognition model is used to perform speech recognition on speech content contained in speech data based on global information and local information of the speech data to obtain a target recognition result of the speech content, and a language type corresponding to the speech content is the target language type.

[0193] Optionally, the processor may also execute program code for the following steps: performing multiple data augmentations on the first speech sample to obtain an augmented sample set; performing semi-supervised learning on the initial recognition model using the first speech sample and the augmented sample set to obtain a semi-supervised loss value; performing self-supervised learning on the initial recognition model using the second speech sample to obtain a self-supervised loss value; obtaining a target loss value corresponding to the initial recognition model based on the self-supervised loss value and the semi-supervised loss value; and adjusting the model parameters of the initial recognition model based on the target loss value to obtain a speech recognition model.

[0194] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: responding to an input command applied to the operating interface, driving the voice acquisition device to acquire voice data from a first object, wherein the voice data contains voice content of a target language type, which is any one of multiple preset language types; responding to a voice recognition command applied to the operating interface, displaying the target recognition result of the voice content on the operating interface, wherein the target recognition result is obtained by inputting the acoustic features into a voice recognition model based on the global and local information of the acoustic features of the voice data, and the acoustic features are obtained by extracting features from the voice data using the voice recognition model. The voice recognition model is obtained by semi-supervised and self-supervised learning using a first voice sample and a second voice sample. The first voice sample contains voice samples of multiple preset language types and does not contain annotation information, while the second voice sample contains language samples of the target language type and annotation information.

[0195] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring speech data obtained from speech acquisition of a first object by calling a first interface, wherein the first interface includes a first parameter, the value of which is speech data, the speech data containing speech content of a target language type, the target language type being any one of multiple preset language types; extracting features from the speech data using a speech recognition model to obtain acoustic features of the speech data; inputting the acoustic features into the speech recognition model based on global and local information of the acoustic features to perform speech recognition, obtaining the target recognition result of the speech content, wherein the speech recognition model is obtained through semi-supervised and self-supervised learning using a first speech sample and a second speech sample, the first speech sample containing speech samples of multiple preset language types and without annotation information, the second speech sample containing language samples of the target language type and annotation information; and outputting the target recognition result by calling a second interface, wherein the second interface includes a second parameter, the value of which is the target recognition result.

[0196] The embodiment of the application is adopted to obtain voice data obtained by voice collection on the first object, wherein the voice data contains voice content of a target language type, and the target language type is any one of multiple preset language types; a voice recognition model is used to perform feature extraction on the voice data to obtain acoustic features of the voice data; based on global information and local information of the acoustic features, the acoustic features are input to the voice recognition model for voice recognition to obtain a target recognition result of the voice content, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of multiple preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information, thereby improving the accuracy of voice data recognition; in the application, the voice recognition model can be trained by using multi-task semi-supervised and self-supervised learning, the generalization ability of the voice recognition model can be improved, the voice recognition model can also perform high-precision recognition on voice content of a language type with a small number of samples, thereby improving the recognition accuracy of voice data, and further solving the technical problem of low recognition accuracy of the voice recognition model in the related art.

[0197] Those skilled in the art can understand that Figure 11 The structure shown is only schematic, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. Figure 11 This does not limit the structure of the electronic device. For example, the computer terminal A can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure. Figure 11 Figure 11 The structure shown is only schematic, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. Figure 11 This does not limit the structure of the electronic device. For example, the computer terminal A can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a different configuration from that shown in the figure.

[0198] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by programs instructing the relevant hardware of the terminal device, and the programs can be stored in a computer-readable storage medium, which can include a flash disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0199] Embodiment 10

[0200] The embodiment of the application also provides a storage medium. Optionally, in the embodiment, the above storage medium can be used to save the program code executed by the voice recognition method provided in Embodiment 1.

[0201] Optionally, in the embodiment, the storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0202] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining voice data obtained by voice collection on the first object, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; performing feature extraction on the voice data by using a voice recognition model to obtain acoustic features of the voice data; inputting the acoustic features into the voice recognition model based on global information and local information of the acoustic features to perform voice recognition, and obtaining a target recognition result of the voice content, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning using first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information.

[0203] Optionally, the storage medium is further configured to store program code for performing the following steps: performing multiple data augmentations on the first voice samples to obtain an augmented sample set; performing semi-supervised learning on the voice recognition model using the first voice samples and the augmented sample set to obtain a semi-supervised loss value; performing self-supervised learning on the voice recognition model using the second voice samples to obtain a self-supervised loss value; obtaining a target loss value corresponding to the voice recognition model based on the self-supervised loss value and the semi-supervised loss value; and adjusting model parameters of the voice recognition model based on the target loss value.

[0204] Optionally, the storage medium is further configured to store program code for performing the following steps: performing semi-supervised learning on the voice recognition model using the first voice samples to obtain a first loss value of the voice recognition model; performing semi-supervised learning on the voice recognition model using the first voice samples and the augmented sample set to obtain a second loss value of the voice recognition model; and obtaining a weighted sum of the first loss value and the second loss value to obtain the semi-supervised loss value.

[0205] Optionally, the storage medium is further configured to store program code for performing the following steps: performing feature extraction on the first voice samples to obtain first sample features of the first voice samples; performing masking on part of the first sample features to obtain first mask features; performing quantization on the first sample features to obtain quantized features; and obtaining the first loss value based on the first mask features and the quantized features.

[0206] Optionally, the storage medium is further configured to store program code for performing the following steps: determining positive sample features and negative sample features corresponding to the first mask feature in the quantified features, wherein the positive sample features and the first mask feature correspond to the same first speech sample, and the negative sample features and the first mask feature correspond to different first speech samples; obtaining similarities of the first mask feature with the positive sample features and the negative sample features respectively, to obtain a first similarity and a second similarity; obtaining a sample pair importance factor of the first mask feature and the negative sample features; and obtaining the first loss value based on the first similarity, the second similarity, and the sample pair importance factor.

[0207] Optionally, the storage medium is further configured to store program code for performing the following steps: performing feature extraction on the first speech sample and the enhanced sample set respectively to obtain first sample features of the first speech sample and an enhanced sample feature set of the enhanced sample set; performing speech recognition on the first sample features and the enhanced sample feature set respectively by using the speech recognition model to obtain a first sample recognition result of the first speech sample and an enhanced sample recognition result set of the enhanced sample set; and obtaining the second loss value based on the first sample recognition result and the enhanced sample recognition result set.

[0208] Optionally, the storage medium is further configured to store program code for performing the following steps: processing each enhanced sample recognition result in the first sample recognition result and the enhanced sample recognition result set by using a cross-entropy loss to obtain an original loss value set corresponding to the enhanced sample set; and obtaining the second loss value based on a maximum loss value in the original loss value set.

[0209] Optionally, the storage medium is further configured to store program code for performing the following steps: performing feature extraction on the second speech sample to obtain second sample features; performing masking on part of the second sample features to obtain a second mask feature; performing speech recognition on the second mask feature by using the speech recognition model to obtain a second sample recognition result of the second speech sample; and obtaining the self-supervised loss value based on the second sample recognition result and the labeled information.

[0210] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining the first speech samples and the second speech samples, wherein the first speech samples contain speech samples of a plurality of preset language types and do not contain annotation information, and the second speech samples contain language samples of a target language type and annotation information, the target language type being any one of the plurality of preset language types; performing semi-supervised and self-supervised learning on the initial recognition model by using the first speech samples and the second speech samples to obtain the speech recognition model, wherein the speech recognition model is configured to perform speech recognition on speech content contained in speech data based on global information and local information of the speech data to obtain a target recognition result of the speech content, and the language type corresponding to the speech content is the target language type.

[0211] Optionally, the storage medium is further configured to store program code for performing the following steps: performing multiple data augmentations on the first speech samples to obtain an augmented sample set; performing semi-supervised learning on the initial recognition model by using the first speech samples and the augmented sample set to obtain a semi-supervised loss value; performing self-supervised learning on the initial recognition model by using the second speech samples to obtain a self-supervised loss value; obtaining a target loss value corresponding to the initial recognition model based on the self-supervised loss value and the semi-supervised loss value; and adjusting model parameters of the initial recognition model based on the target loss value to obtain the speech recognition model.

[0212] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: in response to an input instruction acting on the operation interface, driving the speech collection device to collect speech of the first object to obtain speech data, wherein the speech data contains speech content of a target language type, and the target language type is any one of a plurality of preset language types; and in response to a speech recognition instruction acting on the operation interface, displaying a target recognition result of the speech content on the operation interface, wherein the target recognition result is obtained by inputting acoustic features of the speech data into the speech recognition model for speech recognition, the acoustic features are obtained by performing feature extraction on the speech data by using the speech recognition model, and the speech recognition model is obtained by performing semi-supervised and self-supervised learning on the first speech samples and the second speech samples, the first speech samples contain speech samples of a plurality of preset language types and do not contain annotation information, and the second speech samples contain language samples of the target language type and annotation information.

[0213] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining voice data obtained by voice collection on the first object by calling the first interface, wherein the first interface comprises a first parameter, a parameter value of the first parameter is the voice data, the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; performing feature extraction on the voice data by using a voice recognition model to obtain acoustic features of the voice data; inputting the acoustic features into the voice recognition model based on global information and local information of the acoustic features to perform voice recognition to obtain a target recognition result of the voice content, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning of first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information; and outputting the target recognition result by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the target recognition result.

[0214] By adopting the embodiment of the application, voice data obtained by voice collection on the first object is obtained, wherein the voice data contains voice content of a target language type, and the target language type is any one of a plurality of preset language types; feature extraction is performed on the voice data by using a voice recognition model to obtain acoustic features of the voice data; the acoustic features are input into the voice recognition model based on global information and local information of the acoustic features to perform voice recognition to obtain a target recognition result of the voice content, wherein the voice recognition model is obtained by semi-supervised and self-supervised learning of first voice samples and second voice samples, the first voice samples contain voice samples of a plurality of preset language types and do not contain labeled information, and the second voice samples contain language samples of the target language type and labeled information, thereby improving the accuracy of voice data recognition is realized; in the application, the voice recognition model can be trained by using multi-task semi-supervised and self-supervised learning, the generalization ability of the voice recognition model can be improved, the voice recognition model can also perform high-precision recognition on voice content of a language type with a small number of samples, thereby the recognition accuracy of voice data can be improved, and the technical problem of low recognition accuracy of the voice recognition model in the related art is solved.

[0215] The serial numbers of the above embodiments of the application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0216] In the above embodiments of the application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0217] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented by other ways. Among them, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0218] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0219] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0220] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0221] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A speech recognition method, characterized in that, include: Acquire speech data obtained by voice acquisition of a first object, wherein the speech data contains speech content of a target language type, and the target language type is any one of a variety of preset language types; The speech data is subjected to feature extraction using a speech recognition model to obtain the acoustic features of the speech data; Based on the global and local information of the acoustic features, the acoustic features are input into the speech recognition model for speech recognition to obtain the target recognition result of the speech content. The speech recognition model is obtained by semi-supervised and self-supervised learning using a first speech sample and a second speech sample. The first speech sample contains speech samples of the multiple preset language types and does not contain annotation information. The second speech sample contains language samples of the target language type and annotation information. The model parameters of the speech recognition model are adjusted by a target loss value, which is obtained based on a self-supervised loss value and a semi-supervised loss value. The semi-supervised loss value is obtained by performing semi-supervised learning on the speech recognition model using a first speech sample and an augmented sample set. The augmented sample set is obtained by performing multiple data augmentations on the first speech sample.

2. The method according to claim 1, characterized in that, The method further includes: The first speech sample is augmented multiple times to obtain an augmented sample set. The speech recognition model is semi-supervised using the first speech sample and the enhanced sample set to obtain a semi-supervised loss value. The speech recognition model is self-supervised learning is performed using the second speech sample to obtain the self-supervised loss value; Based on the self-supervised loss value and the semi-supervised loss value, the target loss value corresponding to the speech recognition model is obtained; The model parameters of the speech recognition model are adjusted based on the target loss value.

3. The method according to claim 2, characterized in that, The speech recognition model is semi-supervised using the first speech sample and the enhanced sample set to obtain the semi-supervised loss value, which includes: The speech recognition model is semi-supervised using the first speech sample to obtain the first loss value of the speech recognition model. The speech recognition model is semi-supervised using the first speech sample and the enhanced sample set to obtain the second loss value of the speech recognition model. The weighted sum of the first loss value and the second loss value is obtained to obtain the semi-supervised loss value.

4. The method according to claim 3, characterized in that, Using the first speech sample to perform semi-supervised learning on the speech recognition model, the first loss value of the speech recognition model is obtained, including: Feature extraction is performed on the first speech sample to obtain the first sample feature of the first speech sample; By masking some features in the first sample features, a first mask feature is obtained; The features of the first sample are quantized to obtain quantized features; The first loss value is obtained based on the first mask feature and the quantization feature.

5. The method according to claim 4, characterized in that, Based on the first mask feature and the quantization feature, the first loss value is obtained as follows: Determine the positive sample feature and negative sample feature corresponding to the first mask feature in the quantization features, wherein the positive sample feature and the first mask feature correspond to the same first speech sample, and the negative sample feature and the first mask feature correspond to different first speech samples; The similarity between the first mask feature and the positive sample feature and the negative sample feature is obtained respectively to obtain the first similarity and the second similarity; Obtain the sample pair importance factor of the first mask feature and the negative sample feature; The first loss value is obtained based on the first similarity, the second similarity, and the importance factor of the sample pair.

6. The method according to claim 3, characterized in that, Semi-supervised learning is performed on the speech recognition model using the first speech sample and the enhanced sample set to obtain the second loss value of the speech recognition model, including: Feature extraction is performed on the first speech sample and the enhanced sample set respectively to obtain the first sample feature of the first speech sample and the enhanced sample feature set of the enhanced sample set; The speech recognition model is used to perform speech recognition on the first sample features and the enhanced sample feature set respectively, to obtain the first sample recognition result of the first speech sample and the enhanced sample recognition result set of the enhanced sample set; The second loss value is obtained based on the first sample recognition result and the enhanced sample recognition result set.

7. The method according to claim 6, characterized in that, Based on the first sample recognition result and the enhanced sample recognition result set, the second loss value is obtained as follows: The cross-entropy loss is used to process each enhanced sample recognition result in the first sample recognition result and the enhanced sample recognition result set to obtain the original loss value set corresponding to the enhanced sample set; The second loss value is obtained based on the maximum loss value in the original set of loss values.

8. The method according to claim 2, characterized in that, The speech recognition model is self-supervised using the second speech sample to obtain the self-supervised loss value, which includes: Feature extraction is performed on the second speech sample to obtain the second sample features; By masking some features in the second sample features, a second mask feature is obtained; The speech recognition model is used to perform speech recognition on the second mask features to obtain the second sample recognition result of the second speech sample; Based on the second sample identification result and the annotation information, the self-supervised loss value is obtained.

9. A model training method, characterized in that, include: Acquire a first speech sample and a second speech sample, wherein the first speech sample contains speech samples of multiple preset language types and does not contain annotation information, and the second speech sample contains language samples of a target language type and annotation information, wherein the target language type is any one of the multiple preset language types; The initial recognition model is semi-supervised and self-supervised learning is performed using the first speech sample and the second speech sample to obtain a speech recognition model. The speech recognition model is used to perform speech recognition on the speech content contained in the speech data based on the global and local information of the speech data to obtain the target recognition result of the speech content. The language type corresponding to the speech content is the target language type. The model parameters of the speech recognition model are adjusted by a target loss value, which is obtained based on a self-supervised loss value and a semi-supervised loss value. The semi-supervised loss value is obtained by performing semi-supervised learning on the speech recognition model using a first speech sample and an augmented sample set. The augmented sample set is obtained by performing multiple data augmentations on the first speech sample.

10. The method according to claim 9, characterized in that, Using the first and second speech samples, the initial recognition model is subjected to semi-supervised and self-supervised learning to obtain a speech recognition model including: The first speech sample is augmented multiple times to obtain an augmented sample set. The initial recognition model is semi-supervised learning is performed using the first speech sample and the enhanced sample set to obtain a semi-supervised loss value. The initial recognition model is subjected to self-supervised learning using the second speech sample to obtain a self-supervised loss value. Based on the self-supervised loss value and the semi-supervised loss value, the target loss value corresponding to the initial recognition model is obtained; The model parameters of the initial recognition model are adjusted based on the target loss value to obtain the speech recognition model.

11. A speech recognition method, characterized in that, include: In response to an input command applied to the operating interface, the voice acquisition device is driven to acquire voice data from the first object, wherein the voice data contains voice content of a target language type, and the target language type is any one of a variety of preset language types. In response to a speech recognition command applied to the operation interface, the target recognition result of the speech content is displayed on the operation interface. The target recognition result is obtained by inputting the acoustic features into a speech recognition model for speech recognition based on the global and local information of the acoustic features of the speech data. The acoustic features are obtained by extracting features from the speech data using the speech recognition model. The speech recognition model is obtained by semi-supervised and self-supervised learning using a first speech sample and a second speech sample. The first speech sample contains speech samples of the multiple preset language types and does not contain annotation information. The second speech sample contains language samples of the target language type and annotation information. The model parameters of the speech recognition model are adjusted by a target loss value, which is obtained based on a self-supervised loss value and a semi-supervised loss value. The semi-supervised loss value is obtained by performing semi-supervised learning on the speech recognition model using a first speech sample and an augmented sample set. The augmented sample set is obtained by performing multiple data augmentations on the first speech sample.

12. A speech recognition method, characterized in that, include: The voice data obtained by voice acquisition of the first object is obtained by calling the first interface. The first interface includes a first parameter, the parameter value of the first parameter is the voice data, and the voice data contains voice content of a target language type. The target language type is any one of a variety of preset language types. The speech data is subjected to feature extraction using a speech recognition model to obtain the acoustic features of the speech data; Based on the global and local information of the acoustic features, the acoustic features are input into the speech recognition model for speech recognition to obtain the target recognition result of the speech content. The speech recognition model is obtained by semi-supervised and self-supervised learning using a first speech sample and a second speech sample. The first speech sample contains speech samples of the multiple preset language types and does not contain annotation information. The second speech sample contains language samples of the target language type and annotation information. The target recognition result is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target recognition result; The model parameters of the speech recognition model are adjusted by a target loss value, which is obtained based on a self-supervised loss value and a semi-supervised loss value. The semi-supervised loss value is obtained by performing semi-supervised learning on the speech recognition model using a first speech sample and an augmented sample set. The augmented sample set is obtained by performing multiple data augmentations on the first speech sample.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 12.

14. A computer terminal, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 12.

Images