Speech recognition method, device, electronic device and storage medium

Abstract

The present application discloses a speech recognition method, device, electronic device and storage medium. When performing speech recognition on the speech data of the target object, the domain information of the speech data is obtained synchronously. Based on the domain information, a second speech recognition model adapted to the domain of the speech data can be determined, thereby improving the problem of incomplete correspondence between the domain of the speech data and the model for processing the speech data in the speech recognition application; and in the specific recognition process, the speech data is first recognized by the basic first speech recognition model to obtain the initial text data; subsequently, according to the determined second speech recognition model, the feature information of the speech data and the recognized initial text data is combined to perform deep-level recognition of features in both speech and text modes, thereby obtaining the target text data, so as to effectively improve the accuracy of the speech data recognition results. The technical solution of the present application can be applied to the field of speech recognition technology.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, more and more related applications have been developed, bringing a lot of convenience to people's work and study. Among them, automatic speech recognition technology is a technology that can recognize and convert input voice data into text data. It has been widely used in various fields such as industry, home appliances, communications, medical services and electronic products.

[0003] In related technologies, speech recognition technology is often implemented based on machine learning models, which extract feature information of speech data through the model to identify text content. In actual applications, it is found that the training data used in the model training process and the data received during actual operation may not fully correspond to each other in the field. For example, the model is mainly trained based on adult speech data. When faced with the recognition needs of children's speech data, there is a problem of low recognition accuracy. Summary of the invention

[0004] The embodiments of the present application provide a speech recognition method, device, electronic device and storage medium, which can improve the accuracy of speech recognition.

[0005] On the one hand, an embodiment of the present application provides a speech recognition method, the method comprising:

[0006] Acquire speech data of a target object and domain information of the speech data;

[0007] Inputting the speech data into a first speech recognition model for speech recognition processing to obtain initial text data;

[0008] Determining, according to the domain information, a second speech recognition model adapted to the domain of the speech data;

[0009] The first feature information of the speech data and the second feature information of the initial text data are extracted by the second speech recognition model, and speech recognition processing is performed based on the first feature information and the second feature information to obtain target text data.

[0010] On the other hand, an embodiment of the present application further provides a speech recognition device, the device comprising:

[0011] An acquisition module, used to acquire the voice data of a target object and the domain information of the voice data;

[0012] A first recognition module, used for inputting the speech data into a first speech recognition model for speech recognition processing to obtain initial text data;

[0013] An adaptation module, configured to determine, based on the domain information, a second speech recognition model adapted to the domain of the speech data;

[0014] The second recognition module is used to extract the first feature information of the speech data and the second feature information of the initial text data through the second speech recognition model, and perform speech recognition processing based on the first feature information and the second feature information to obtain target text data.

[0015] Furthermore, the first speech recognition model includes an audio encoder and a first text decoder; the first recognition module includes:

[0016] An audio encoding submodule, used for inputting the speech data into the first speech recognition model, encoding the speech data through the audio encoder, and obtaining third feature information of the speech data;

[0017] The first text decoding submodule is used to decode the third feature information through the first text decoder to obtain the initial text data.

[0018] Furthermore, the second speech recognition model includes a speech adaptation sub-model and a text adaptation sub-model; the second recognition module includes:

[0019] A first feature extraction submodule, used for extracting first feature information of the speech data through the speech adaptation submodel;

[0020] A second feature extraction submodule, used for extracting second feature information of the initial text data through the text adaptation submodel;

[0021] A feature fusion submodule, used for fusing the first feature information and the second feature information to obtain fused feature information;

[0022] The recognition submodule is used to obtain the target text data according to the fused feature information.

[0023] Furthermore, the device further comprises a first training module, wherein the first training module comprises:

[0024] A first acquisition submodule is used to acquire first speech sample data of a first domain, first text label data of the first speech sample data, and first text recognition data of the first speech sample data; the first text label data is used to characterize the real text content of the first speech sample data, and the first text recognition data is obtained by performing speech recognition processing on the first speech sample data by using a trained first speech recognition model;

[0025] A first processing submodule, configured to input the first speech sample data and the first text recognition data into an initialized second speech recognition model for speech recognition processing, and obtain second text recognition data output by the second speech recognition model;

[0026] A first loss value calculation submodule, used to determine a first loss value of training according to the second text recognition data and the first text label data;

[0027] The first updating submodule is used to update the parameters of the second speech recognition model according to the first loss value to obtain a trained second speech recognition model.

[0028] Furthermore, the device further comprises an amplification module, wherein the amplification module comprises:

[0029] A selection submodule, used for randomly selecting original text data from the first text label data or the first text recognition data;

[0030] A transformation submodule, used for transforming the original text data to construct virtual text recognition data;

[0031] The data adding submodule is used to amplify the first text recognition data by using the virtual text recognition data.

[0032] Furthermore, the above transformation submodule is specifically used for:

[0033] Performing word segmentation processing on the original text data to obtain an original word segmentation phrase corresponding to the original text data; the original word segmentation phrase includes a plurality of words;

[0034] Transforming the original segmented word phrase to obtain a transformed target segmented word phrase;

[0035] The virtual text recognition data is constructed based on the target segmented word phrase.

[0036] Further, the above-mentioned transformation submodule is specifically used to perform at least one of the following steps:

[0037] Deleting at least one of the words;

[0038] Alternatively, at least one of the words is replaced;

[0039] Alternatively, a preset word is inserted between at least two adjacent words.

[0040] Furthermore, the above transformation submodule is specifically used for:

[0041] Extracting first audio feature information of the word to be replaced;

[0042] Perform a matching search in the dictionary to determine a target word; wherein the similarity between the second audio feature information of the target word and the first audio feature information is higher than a preset threshold;

[0043] The target word is used to replace the word to be replaced.

[0044] Furthermore, the device further comprises a second training module, wherein the second training module comprises:

[0045] A second processing submodule, configured to input the first speech sample data into the speech adaptation submodel for speech recognition processing to obtain third text recognition data;

[0046] A second loss value calculation submodule, used to determine a second loss value of training according to the third text recognition data and the first text label data;

[0047] A second updating submodule, used for updating the parameters of the speech adaptation submodel according to the second loss value to obtain a trained speech adaptation submodel;

[0048] A third processing submodule, configured to input the first text recognition data into the text adaptation submodel to perform domain adaptation processing on the text content, thereby obtaining fourth text recognition data;

[0049] A third loss value calculation submodule, used to determine a third loss value of training according to the fourth text recognition data and the first text label data;

[0050] A third updating submodule, used for updating the parameters of the text adaptation submodel according to the third loss value to obtain a trained text adaptation submodel;

[0051] The initialization submodule is used to obtain an initialized second speech recognition model based on the trained speech adaptation submodel and the trained text adaptation submodel.

[0052] Further, the text adaptation sub-model includes a text encoder and a second text decoder; the third processing sub-module is specifically used for:

[0053] Inputting the first text recognition data into the text adaptation sub-model, encoding the first text recognition data through the text encoder, and obtaining fourth feature information of the first text recognition data;

[0054] The fourth feature information is decoded by the second text decoder to obtain the fourth text recognition data.

[0055] Furthermore, the device also includes:

[0056] A second acquisition module is used to acquire second voice sample data of a second field and second text label data of the second voice sample data; the second text label data is used to represent the real text content of the second voice sample data;

[0057] A third training module, used for training the first speech recognition model by using the second speech sample data and the second text label data to obtain a trained first speech recognition model;

[0058] A construction module is used to construct an initialized speech adaptation sub-model according to the model structure and model parameters of the trained first speech recognition model.

[0059] On the other hand, an embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned speech recognition method when executing the computer program.

[0060] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above-mentioned speech recognition method.

[0061] On the other hand, the embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device executes the above-mentioned speech recognition method.

[0062] The embodiments of the present application include at least the following beneficial effects: when performing speech recognition on the speech data of the target object, the embodiments of the present application synchronously obtains the domain information of the speech data, and based on the domain information, can determine a second speech recognition model adapted to the domain of the speech data, thereby being able to improve the problem of incomplete correspondence between the domains of the speech data and the model for processing the speech data in the speech recognition application; and, in a specific recognition process, firstly recognize the speech data through the basic first speech recognition model to obtain initial text data; subsequently, based on the determined second speech recognition model, combined with the feature information of the speech data and the initial text data that has been recognized, deep feature recognition is performed in both speech and text modes, thereby obtaining the target text data, thereby being able to effectively improve the accuracy of the speech data recognition results.

[0063] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0065] Figure 1 A schematic diagram of an implementation environment of a speech recognition method provided in an embodiment of the present application;

[0066] Figure 2 A schematic diagram of an implementation environment of another speech recognition method provided in an embodiment of the present application;

[0067] Figure 3 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0068] Figure 4 A schematic diagram of a model structure of a first speech recognition model provided in an embodiment of the present application;

[0069] Figure 5 A schematic diagram of a model structure of a second speech recognition model provided in an embodiment of the present application;

[0070] Figure 6 A schematic diagram of fusing first feature information and second feature information provided in an embodiment of the present application;

[0071] Figure 7 A schematic diagram of data flow during a training process provided in an embodiment of the present application;

[0072] Figure 8 A schematic diagram of the model structure of a text adapter model provided in an embodiment of the present application;

[0073] Fig. 9 A schematic diagram of WER data corresponding to each identification strategy provided in the embodiment of the present application;

[0074] Fig.10 A schematic diagram of a speech recognition case result corresponding to each recognition strategy provided in the embodiment of the present application;

[0075] Fig.11 A business data flow diagram for an oral test application scenario provided in an embodiment of the present application;

[0076] Fig.12 A schematic diagram of the interface of the oral test APP provided in the embodiment of the present application;

[0077] Fig.13 A schematic diagram of the interface of the video playback software provided in the embodiment of the present application;

[0078] Fig.14 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0079] Fig.15 A structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0080] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0082] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0083] Automatic Speech Recognition (ASR): This technology enables computers to "dictate" continuous speech spoken by different people, commonly known as "voice dictation machine". It is a technology that realizes the conversion of "sound" to "text" and can convert voice information into text information.

[0084] Transfer Learning is a machine learning method that can transfer knowledge from one field (source domain) to another field (target domain), so that the target domain can achieve faster and better learning results.

[0085] Source domain: The knowledge domain where the knowledge to be transferred is located during the transfer learning process. There is a large amount of general knowledge for transfer learning.

[0086] Target domain: The knowledge domain to which the transferred knowledge is to be transferred during the transfer learning process, that is, the domain where the target task in the machine learning application is located.

[0087] WER (word error rate) is an indicator to measure the degree of ASR errors, including insertion, deletion and substitution errors generated after ASR recognition. The larger the WER, the worse the recognition effect; conversely, the smaller the WER, the better the recognition effect.

[0088] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0089] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning (deep learning) usually includes artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0090] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. The natural language involved in this field is the language used by people in daily life, so it is also closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0091] Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer and application service layer. Blockchain can include public chain, alliance chain and private chain. Among them, public chain refers to a blockchain that anyone can enter the blockchain network at any time to read data, send data or compete for accounting; alliance chain refers to a blockchain jointly managed by several organizations or institutions; private chain refers to a blockchain with certain centralized control. The write right of the private chain account book is controlled by a certain organization or institution, and the access and use of data are strictly managed by permissions.

[0092] The speech recognition method provided in the embodiments of the present application involves artificial intelligence technology, which mainly involves natural language processing technology and machine learning / deep learning technology in the field of artificial intelligence technology. Specifically, the method provided in the embodiments of the present application can use natural language processing technology and machine learning technology to process speech data and text data, thereby obtaining more accurate speech recognition results.

[0093] The method provided in the embodiment of the present application can be executed in application scenarios such as data processing and human-computer interaction: for example, in the application scenario of data processing, there is a need for real-time translation and recording of voice data. In this scenario, the recognition method provided in the embodiment of the present application can be used to recognize and process the voice data, thereby converting the voice data into corresponding text data. In the application scenarios of human-computer interaction, such as in intelligent transportation systems and intelligent vehicle-road cooperative systems, by recognizing and analyzing the driver's voice commands, it can help to generate more accurate response information, thereby facilitating the provision of navigation guidance.

[0094] Of course, it should be noted that the above application scenarios are only exemplary and do not limit the actual application of the method in the embodiment of the present application. Those skilled in the art can understand that in different application scenarios, the method provided in the embodiment of the present application can be used to perform the specified task.

[0095] Furthermore, it should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on the target object's information, the target object's behavioral data, the target object's historical data, and the target object's location information and other data related to the target object's identity or characteristics, the target object's permission or consent will be obtained first, and the collection, use and processing of these data will comply with the relevant laws, regulations and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain sensitive information of the target object, the target object's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the target object's separate permission or separate consent, the necessary target object-related data used to enable the normal operation of the embodiment of the present application will be obtained.

[0096] With the development of artificial intelligence technology, more and more related applications have been developed, bringing a lot of convenience to people's work and study. Among them, automatic speech recognition technology is a technology that can recognize and convert input voice data into text data. It has been widely used in various fields such as industry, home appliances, communications, medical services and electronic products.

[0097] In related technologies, speech recognition technology is often implemented based on machine learning models, which extract feature information of speech data through the model to identify text content. In actual applications, it is found that the training data used in the model training process and the data received during actual operation may not fully correspond to each other in the field. For example, the model is mainly trained based on adult speech data. When faced with the recognition needs of children's speech data, there is a problem of low recognition accuracy.

[0098] Based on this, the embodiments of the present application provide a speech recognition method, device, electronic device and storage medium. When performing speech recognition on the speech data of the target object, the domain information of the speech data is synchronously obtained. Based on the domain information, a second speech recognition model adapted to the domain of the speech data can be determined, thereby improving the problem of incomplete correspondence between the domains of the speech data and the model that processes the speech data in the speech recognition application; and, in the specific recognition process, the speech data is first recognized by the basic first speech recognition model to obtain initial text data; subsequently, according to the determined second speech recognition model, combined with the feature information of the speech data and the initial text data that has been recognized, deep feature recognition is performed in both speech and text modes to obtain the target text data. In this way, the accuracy of the speech data recognition results can be effectively improved.

[0099] The specific implementation of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0100] Reference Figure 1 , Figure 1 A schematic diagram of an implementation environment of the speech recognition method provided in the embodiment of the present application is shown. In the implementation environment, the main hardware and software entities involved include the first terminal 110.

[0101] Specifically, a related application may be installed in the first terminal 110, and the application may include a voice recognition component and a display component, wherein the voice recognition component may execute the voice recognition method provided in the embodiment of the present application based on the voice data acquired by the first terminal 110, thereby obtaining the corresponding voice recognition result, that is, text data containing the voice data content, and displaying it on the application interface through the display component for browsing. It should be noted that, here, the voice data acquired by the first terminal 110 may be voice data collected by other devices and then transmitted to the first terminal 110, or may be collected by the first terminal 110 based on its own voice data acquisition application or voice data acquisition component.

[0102] In addition, refer to Figure 2 , Figure 2 A schematic diagram of another implementation environment of the speech recognition method provided in an embodiment of the present application, wherein the software and hardware entities mainly involved in the implementation environment include a second terminal 210 and a server 220, wherein the second terminal 210 and the server 220 are communicatively connected.

[0103] Specifically, the server 220 may provide a voice recognition service, and the second terminal 210 may be installed with a related application, which may include a communication component and a display component. When the second terminal 210 has a voice recognition demand, the acquired voice data may be transmitted to the server 220 through the communication component, and the server 220 may execute the voice recognition method provided in the embodiment of the present application to obtain the corresponding voice recognition result, that is, text data containing the voice data content, and transmit it to the communication component in the second terminal 210, which may then transmit the text data to the display component so that it is displayed on the application interface for browsing. It should be noted that, here, the voice data acquired by the second terminal 210 may be voice data collected by other devices and then transmitted to the second terminal 210, or may be collected by the second terminal 210 based on its own voice data collection application or voice data collection component.

[0104] Among them, the first terminal 110 and the second terminal 210 of the above embodiments may include a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, but are not limited thereto.

[0105] Server 220 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0106] In addition, the server 220 may also be a node server in the blockchain network.

[0107] The second terminal 210 and the server 220 may establish a communication connection via a wireless network or a wired network. The wireless network or wired network uses standard communication technology and / or protocols, and the network may be set to the Internet or any other network, such as but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network.

[0108] Of course, it is understandable that Figure 1 and Figure 2The implementation environment in the present application is only some optional application scenarios of the speech recognition method provided in the embodiment of the present application, and the actual application is not fixed. Figure 1 and Figure 2 The method provided in the embodiment of the present application can be applied to various technical fields, and the present application does not impose any specific limitation on this.

[0109] Reference Figure 3 , Figure 3 The flowchart of the speech recognition method provided in the embodiment of the present application is as follows. The speech recognition method can be executed by the terminal alone or by the terminal and the server in cooperation. The speech recognition method includes but is not limited to the following steps 301 to 304.

[0110] Step 301: Acquire the speech data of the target object and the domain information of the speech data.

[0111] In this step, the target object may include any person who has a need for voice recognition or other devices and equipment that can output voice data, but is not limited to this, and this application does not make specific limitations on this. When performing voice recognition on the target object, it is necessary to obtain the voice data of the target object. Specifically, in this step, there is no restriction on the channel for obtaining the voice data, which can be directly collected from the environment where the target object is located through a sound receiving device, or obtained from other electronic devices and computer systems through a data transmission interface or remote communication transmission.

[0112] In this step, when the speech data of the target object is subjected to speech recognition, the domain information of the speech data is also obtained. Here, the domain information is used to characterize the knowledge domain corresponding to or covered by the speech data, which belongs to a way of dividing different data in the transfer learning task. In general, when the data with different distributions are mapped to the same feature space, the distance between the features corresponding to the data belonging to the same knowledge domain (domain) in the space will be as close as possible. Specifically, for the speech data in this application, the division type of the corresponding domain information and the data range covered can be flexibly set as needed. For example, in some embodiments, the domain in which the speech data is located can be divided according to the language corresponding to the speech data. In this way, the domain information may include type information such as "English", "French" and "German". In some embodiments, the speech data may be divided according to some attributes of the target object. For example, the domain information of the speech data may be distinguished according to the characteristics of the target object such as gender and age. In the embodiment of the present application, the specific data format of the domain information and the corresponding actual information meaning may be flexibly set as needed. For example, its data format may include at least one of a number, a vector, a matrix or a tensor, and the present application does not limit this.

[0113] It is understandable that, in some embodiments, the domain information of the voice data can be refined by combining multiple division criteria. For example, the domain information corresponding to the voice data of a target object may include information content in multiple dimensions such as language, gender, and age. For example, the content of the domain information may be specifically "adult male, French language". Of course, the specific domain information division criteria can be implemented as needed. Generally speaking, the more detailed the division of the domain information, the more accurate the corresponding voice recognition result.

[0114] It should be noted that in the embodiments of the present application, the data ranges covered by different field information may partially overlap or completely cover each other, and the present application does not impose any restrictions on this.

[0115] Step 302: Input the speech data into the first speech recognition model for speech recognition processing to obtain initial text data.

[0116] In this step, after obtaining the speech data of the target object, the speech data can be input into a speech recognition model for speech recognition processing. The speech recognition model here is recorded as a first speech recognition model. In the embodiment of the present application, there is no specific restriction on the structure of the first speech recognition model, which can be constructed using any commonly used algorithm in the field of machine learning.

[0117] It should be noted that in the field of machine learning, the model needs to be trained before it is put into use to improve the prediction accuracy of the model for input data. For the application field of the present application, the first speech recognition model needs to be trained by batches of speech data and its corresponding real text data to improve the recognition accuracy of the model in converting speech data into text data. However, the field of the first speech recognition model is not constrained in the present application. In other words, the training data used by the first speech recognition model in the training process may not completely correspond to the field of the speech data of the target object. Therefore, as mentioned in the background technology part, the initial text data output by the first speech recognition model may also be inaccurate. Therefore, in the embodiment of the present application, the field adaptation correction is also performed through subsequent steps to improve the recognition accuracy of the obtained text data.

[0118] Step 303: Determine a second speech recognition model adapted to the domain of the speech data based on the domain information.

[0119] In this step, after obtaining the domain information of the speech data of the target object, a speech recognition model adapted to the domain of the speech data can be determined according to the domain information. In the embodiment of the present application, the speech recognition model is recorded as the second speech recognition model. Here, it should be noted that in the present application, multiple second speech recognition models can be established in advance according to the number of domain information, and each domain information corresponds to the second speech recognition model. For each second speech recognition model, the domain adaptation of the speech data refers to the training data corresponding to the domain information of the speech data. From the perspective of transfer learning, in the embodiment of the present application, the first speech recognition model can be regarded as a speech recognition model trained by the data of the source domain, and the second speech recognition model can be regarded as a speech recognition model trained by the data of the target domain. Specifically, for example, in some embodiments, the second speech recognition model includes a speech recognition model trained based on English speech data, a speech recognition model trained based on French speech data, and a speech recognition model trained based on German speech data. When the domain information of a certain speech data is the type information of German, the speech recognition model trained based on German speech data can be determined as the adapted second speech recognition model.

[0120] It should be noted that, in the embodiment of the present application, the number of the second speech recognition models can be exactly the same as the number of the domain information, or can be less than the number of the domain information. In particular, when the number of the second speech recognition models is less than the number of the domain information, the model with the highest degree of matching between the training data and the domain information can be determined as the second speech recognition model adapted to the domain of the speech data.

[0121] Step 304: extracting first feature information of the speech data and second feature information of the initial text data through a second speech recognition model, and performing speech recognition processing based on the first feature information and the second feature information to obtain target text data.

[0122] In this step, after determining the second speech recognition model adapted to the field of the speech data, the second speech recognition model can be used to perform deep speech recognition processing to obtain the target text data, so that the accuracy of the speech data recognition result can be effectively improved. Specifically, for the second speech recognition model used in the embodiment of the present application, it can extract two parts of feature information, wherein the feature information of the first part can be obtained by extracting the original speech data of the target object; the feature information of the second part can be obtained by extracting the initial text data output by the first speech recognition model. In the embodiment of the present application, the feature information of the first part is recorded as the first feature information, and the feature information of the second part is recorded as the second feature information. Similarly, in the embodiment of the present application, the data format of the first feature information and the second feature information may include at least one of a number, a vector, a matrix or a tensor, and the present application does not limit this. Moreover, it can be understood that in the embodiment of the present application, the feature information described later is applicable to the implementation method of the above data format.

[0123] It can be understood that the first feature information is extracted by the second speech recognition model that is adapted to the domain of the speech data. Based on the domain adaptation of the speech modality, the acoustic features can be well adapted, that is, the first feature information can better reflect the speech content with domain characteristics. The second feature information is the feature information further extracted by the second speech recognition model on the original text data recognition result, which can be domain-adapted on the text modality, so that the model can make fine adjustments to the domain where the speech data is located on the basic text content. In addition, in an embodiment of the present application, the above two parts of feature information are combined for speech recognition processing. In this way, based on the combination of adaptive domain features in speech modality and text modality, a better speech recognition effect can be achieved.

[0124] Specifically, in a possible implementation, the first speech recognition model includes an audio encoder and a first text decoder; inputting speech data into the first speech recognition model for speech recognition processing to obtain initial text data, including:

[0125] Inputting the speech data into the first speech recognition model, encoding the speech data through an audio encoder to obtain third feature information of the speech data;

[0126] The third characteristic information is decoded by the first text decoder to obtain initial text data.

[0127] Reference Figure 4In an embodiment of the present application, an optional model structure of a first speech recognition model is proposed, and the model structure belongs to a sequence to sequence model, namely a Seq2seq model. The sequence to sequence model is one of the more commonly used models in sequence tasks today, and it is good at processing a series of data with continuous relationships, such as speech data, text data, and video data. Specifically, the model framework of the sequence to sequence model generally includes two parts, an encoder part and a decoder part. In an embodiment of the present application, the encoder part is recorded as an audio encoder, and the decoder part is recorded as a first text decoder.

[0128] The encoder and decoder in the sequence-to-sequence model can be implemented using a recurrent neural network (RNN), such as the commonly used long-short term memory network (LSTM). Of course, it is understandable that the above encoder and decoder can be flexibly adjusted as needed, and this application does not limit this. Specifically, in an embodiment of the present application, a pre-trained acoustic model Wav2Vec 2.0 is used as an audio encoder, and a BERT model is used as the first text decoder. Wav2vec 2.0 is a model for self-supervised pre-training based on a large amount of unlabeled data. It consists of a multi-layer convolutional feature encoding layer and a multi-layer transformer. By inputting the original audio signal into the audio encoder, an audio representation of each frame can be obtained. The BERT model uses a bidirectional self-attention mechanism, which can better capture the connection between the context in the sentence, and is currently widely used in the field of text processing.

[0129] for Figure 4 The model structure shown, when applied, can input voice data into the first voice recognition model, and the audio encoder can encode the input voice data and extract feature information that can map the approximate content of the voice data, which is recorded as the third feature information in this application.

[0130] Specifically, in some embodiments, the third feature information may include acoustic feature information of speech data, for example, it may be a digital feature of the audio spectrum of speech data. Specifically, some time-frequency points may be selected from the audio spectrum of speech data according to a predetermined rule, and encoded into a digital sequence, which may be used as the third feature information of speech data. Of course, the feature extraction algorithm used in the embodiment of the present application may be selected as needed. For example, in some embodiments, speech data may be framed and windowed according to a certain frame stack to obtain multiple audio frames. Specifically, framing here refers to cutting the entire segment of speech data into multiple segments according to a preset rule, and each segment is an audio frame. Windowing refers to intercepting each audio frame using a preset window function, so that the framed speech data is more coherent and exhibits better periodic function characteristics. The window function may be any one of a rectangular window, a Hamming window, or a Gaussian window. Then, the acquired audio frame is subjected to a fast Fourier transform (FFT) to obtain the spectrum of each audio frame, and the peak signal in each frame spectrum is selected as the digital feature of the spectrum to construct the acoustic feature information of the speech data; in some embodiments, after obtaining the spectrum of the audio frame, each audio frame can also be divided into multiple sub-bands in the frequency domain, and then the difference in energy difference between any two adjacent sub-bands is calculated, and the acoustic feature information of the speech data is constructed based on the obtained difference. The above-mentioned method generally converts the digital features into a hash sequence during encoding, that is, the hash sequence is used as the acoustic feature information of the audio data. In some embodiments, the audio digital features of the speech data can also be directly extracted as the third feature information by an audio encoder composed of a neural network model, etc.

[0131] Of course, the above-mentioned method of extracting the third feature information based on the audio spectrum is only used for illustration and does not mean to limit the actual implementation of the present application. For example, the present application can also extract the third feature information based on the dimensions of pronunciation accuracy, fluency, rhythm, signal-to-noise ratio, sound intensity, etc. Moreover, in some embodiments, the third feature information extracted from multiple dimensions can be integrated to obtain new third feature information. For example, the third feature information in the form of multiple vectors can be spliced ​​to obtain a new third feature information in the form of a vector.

[0132] After obtaining the third feature information, the third feature information can be decoded by the first text decoder, and the first text decoder can restore the third feature information encoded by the audio encoder into corresponding text sequence data, that is, initial text data.

[0133] In the actual processing process, Figure 4 Taking the model structure shown in as an example, the speech data includes multiple groups of audio data (x1, x2...x m, m is a positive integer), and after inputting it into the audio encoder, the third feature information obtained by the audio encoder includes multiple fixed-length vectors. These fixed-length vectors are sequentially passed to the first text decoder, which uses the vector as the hidden layer input and a start flag as the input of the current position; the output vectors (such as y1, y2...y k , k is a positive integer) respectively corresponds to a unit of text data (generally corresponding to a word). And the first text encoder will pass the output vector of the mapping output to the next unit node, thereby completing the sequence output of the entire initial text data. Based on the output vectors y1, y2...y k The corresponding initial text data can be obtained through dictionary translation and other methods.

[0134] Specifically, in a possible implementation, the second speech recognition model includes a speech adaptation sub-model and a text adaptation sub-model; extracting first feature information of speech data and second feature information of initial text data through the second speech recognition model, and performing speech recognition processing based on the first feature information and the second feature information to obtain target text data, including:

[0135] Extracting first feature information of the speech data through the speech adaptation sub-model;

[0136] Extracting second feature information of the initial text data through a text adaptation sub-model;

[0137] Fusing the first feature information and the second feature information to obtain fused feature information;

[0138] According to the fused feature information, the target text data is obtained.

[0139] Reference Figure 5 In the embodiment of the present application, an optional model structure of a second speech recognition model is provided. Figure 5 In the second speech recognition model shown, it includes a speech adaptation sub-model 510 and a text adaptation sub-model 520. Among them, the speech adaptation sub-model 510 is used to extract the first feature information of the speech data, and the text adaptation sub-model 520 is used to extract the second feature information of the initial text data. The speech adaptation sub-model 510 and the text adaptation sub-model 520 can both be implemented using a sequence-to-sequence model, and this application does not make any specific restrictions on this.

[0140] When using the second speech recognition model for speech recognition, the first feature information and the second feature information can be fused to obtain fused feature information. Here, the method of fusing the first feature information and the second feature information may include data splicing, data weighting, etc., and this application does not impose specific restrictions on this. Then, based on the fused feature information, the corresponding target text data can be obtained through the full connection layer and the Softmax layer mapping. In this way, based on the fusion of adaptive domain features in speech modality and text modality, better speech recognition effect can be achieved.

[0141] Reference Figure 6 , Figure 6 A specific flow chart of fusing the first feature information and the second feature information is shown. Figure 6 It can be seen that in the embodiment of the present application, when the first feature information and the second feature information are fused, the fusion can be performed at the granularity of each word. Specifically, the first feature information output by the first text encoder can be numbered according to the position of the corresponding word in the text data. Similarly, the second feature information output by the second text encoder is also numbered. Then, when performing feature fusion, the first feature information and the second feature information with the same number are fused, and for each fused feature information, an independent Softmax layer mapping is used to obtain the corresponding target text data.

[0142] It should be noted that in the embodiment of the present application, the first feature information may be any feature information obtained during the process of the speech adaptation sub-model processing the input speech data, and is not limited to using the output of the first text decoder as the first feature information. Similarly, the second feature information may also be any feature information obtained during the process of the text adaptation sub-model processing the input initial text data, and is not limited to using the output of the second text decoder as the second feature information.

[0143] The above content introduces the data processing flow of the second speech recognition model provided in the embodiment of the present application for speech recognition. In the field of machine learning, the model generally needs to be trained and adjusted before it is put into use. Therefore, in the embodiment of the present application, a model training method is also provided, which includes the training process of the first speech recognition model and the second speech recognition model. Figure 7 , Figure 7A schematic diagram of the data flow during the overall training process of the embodiment of the present application is shown. Specifically, in the embodiment of the present application, the training data mainly includes speech sample data and text label data. Based on the training data, the first speech recognition model can be trained to obtain the initial text data output by the model; based on the speech sample data and the text label data, the speech adaptation sub-model responsible for speech modality domain adaptation in the second speech recognition model can be trained, and based on the text label data and the initial text data, the text adaptation sub-model responsible for text modality domain adaptation in the second speech recognition model can be trained. In addition, the feature fusion capability of the second speech recognition model can also be trained based on the speech sample data, the initial text data and the text label data, so as to improve the accuracy of the target text data output by the second speech recognition model.

[0144] Specifically, in the embodiment of the present application, in order to improve the training efficiency and the recognition effect of the trained second speech recognition model, the training process of the second speech recognition model can be implemented by a two-stage network training method. In the first stage, the speech adaptation sub-model and the text adaptation sub-model in the second speech recognition model are mainly trained to improve the effect of extracting feature information of both; the second stage trains the second speech recognition model as a whole to improve the effect of the second speech recognition model on the fusion of feature information, thereby improving the speech recognition ability of the model.

[0145] Below, the two-stage training process of the second speech recognition model in the embodiment of the present application is described in detail.

[0146] In the embodiment of the present application, the second speech recognition model is firstly trained in the first stage. The training process of the first stage can be recorded as initialization training. The initialization training process may include:

[0147] Acquire first speech sample data of a first domain, first text label data of the first speech sample data, and first text recognition data of the first speech sample data; the first text label data is used to characterize the real text content of the first speech sample data, and the first text recognition data is obtained by performing speech recognition processing on the first speech sample data by using a trained first speech recognition model;

[0148] Inputting the first speech sample data into the speech adaptation sub-model for speech recognition processing to obtain third text recognition data;

[0149] Determine a second loss value for training according to the third text recognition data and the first text label data;

[0150] According to the second loss value, the parameters of the speech adaptation sub-model are updated to obtain a trained speech adaptation sub-model;

[0151] Inputting the first text recognition data into the text adaptation sub-model to perform domain adaptation processing on the text content to obtain fourth text recognition data;

[0152] Determining a third loss value for training according to the fourth text recognition data and the first text label data;

[0153] According to the third loss value, the parameters of the text adaptation sub-model are updated to obtain a trained text adaptation sub-model;

[0154] Based on the trained speech adaptation sub-model and the trained text adaptation sub-model, an initialized second speech recognition model is obtained.

[0155] In the embodiment of the present application, when the second speech recognition model is trained, the first speech sample data of the first field, the first text label data of the first speech sample data, and the first text recognition data of the first speech sample data can be obtained. Here, the first field refers to the field range of the speech data that can be processed by the second speech recognition model later, and the size of the data range covered can be set according to the needs, and the present application is not limited. It should be noted that in the embodiment of the present application, in actual application, multiple different second speech recognition models can be trained, and each second speech recognition model corresponds to a different first field. Here, different first fields refer to the data range covered by their fields is not exactly the same. In other words, different first fields may have partially overlapping or completely covered data ranges, and the present application is not limited to this. The speech data in the first field is recorded as the first speech sample data, and the first speech sample data can be obtained according to the scope of the first field. For example, when the first field is speech data in the English field, a batch of English speech data can be obtained as the first speech sample data; when the first field is children's speech data, a batch of children's speech data can be obtained as the first speech sample data. In an embodiment of the present application, first text label data and first text recognition data are also obtained synchronously, wherein the first text label data is used to characterize the actual text content of the first voice sample data, and the first text recognition data is obtained by inputting the first voice sample data into a trained first voice recognition model.

[0156] After obtaining the above-mentioned training data, the first speech sample data can be input into the speech adaptation sub-model for speech recognition processing. The speech recognition processing process of the speech adaptation sub-model can be similar to the aforementioned first speech recognition model, and this application will not go into details here. The text data output by the speech adaptation sub-model is recorded as the third text recognition data. In an embodiment of the present application, the accuracy of the speech adaptation sub-model recognition can be determined based on the third text recognition data and the first text label data. Specifically, the deviation between the third text recognition data and the first text label data can be determined to obtain the loss value corresponding to the speech adaptation sub-model, which is recorded as the second loss value. After obtaining the second loss value, the accuracy of the speech adaptation sub-model recognition can be evaluated according to the size of the second loss value, so as to perform back-propagation training on the speech adaptation sub-model and update the relevant parameters inside it.

[0157] Specifically, for a machine learning model, the accuracy of its prediction can be measured by a loss function (LossFunction). The loss function is defined on a single training data and is used to measure the prediction error of a training data. Specifically, the loss value of the training data is determined by the label of the single training data and the prediction result of the model for the training data. In actual training, a training data set has a lot of training data, so the cost function (CostFunction) is generally used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average value of the prediction error of all training data, which can better measure the prediction effect of the model. For a general machine learning model, based on the aforementioned cost function, plus a regularization term that measures the complexity of the model, it can be used as the objective function of the training. Based on this objective function, the loss value of the entire training data set can be calculated. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions of machine learning models, and will not be elaborated one by one here.

[0158] For the speech adaptation sub-model, the single training data is the first speech sample data, the label of the first speech sample data is the first text label data, and the prediction result of the model for the training data is the third text recognition data. In the embodiment of the present application, one loss function can be selected from the aforementioned loss function types to determine the loss value of the training, that is, the second loss value between the first text label data and the third text recognition data. For example, the second loss value can be calculated by the following cross entropy loss function:

[0159]

[0160] Where, L fuserepresents the second loss value, i represents the word number in the first text label data, n represents the number of words in the first text label data, Represents the label of the i-th word in the first text label data; Represents the predicted probability distribution of the i-th word in the third text recognition data output by the speech adaptation sub-model.

[0161] Based on the second loss value of the training, the back propagation algorithm is used to update the parameters of the speech adaptation sub-model, and the trained speech adaptation sub-model can be obtained by iterating the preset rounds.

[0162] Similarly, in an embodiment of the present application, for the training process of the text adaptation sub-model, the first text recognition data can be input into the text adaptation sub-model to perform field adaptation processing on the text content. The text data output by the text adaptation sub-model is recorded as the fourth text recognition data. According to the fourth text recognition data and the first text label data, the accuracy of the text adaptation sub-model in adapting the text content can be determined. Specifically, the deviation between the fourth text recognition data and the first text label data can be determined to obtain the loss value corresponding to the text adaptation sub-model, which is recorded as the third loss value. After obtaining the third loss value, the accuracy of the text adaptation sub-model in adapting the text content can be evaluated according to the size of the third loss value, so as to perform back-propagation training on the text adaptation sub-model and update the relevant parameters inside it. The specific training process is similar to the aforementioned speech adaptation sub-model, which will not be repeated here.

[0163] Specifically, in a possible implementation, the text adaptation sub-model includes a text encoder and a second text decoder; the first text recognition data is input into the text adaptation sub-model to perform domain adaptation processing of the text content, and the fourth text recognition data is obtained, including:

[0164] Inputting the first text recognition data into the text adaptation sub-model, encoding the first text recognition data through a text encoder, and obtaining fourth feature information of the first text recognition data;

[0165] The fourth feature information is decoded by the second text decoder to obtain fourth text recognition data.

[0166] Reference Figure 8 In the embodiment of the present application, an optional model structure of a text adaptation sub-model is proposed, and the model structure also belongs to a sequence-to-sequence model. In the embodiment of the present application, the encoder part is recorded as a text encoder, and the decoder part is recorded as a second text decoder.

[0167] Similarly, the text encoder and the second text decoder in the text adaptation sub-model can also be implemented using a recurrent neural network (RNN), and can be flexibly adjusted as needed, and this application does not limit this. Specifically, in an embodiment of this application, a BERT model is used as the text encoder and the second text decoder.

[0168] for Figure 8 The model structure shown, when applied, can input the first text recognition data into a text encoder, and the text encoder can encode the input first text recognition data and extract feature information that can map the approximate content of the first text recognition data, which is recorded as the fourth feature information in this application.

[0169] Specifically, in some embodiments, if the data format of the first text recognition data is a text format, it can be encoded and converted by inputting it into a text encoder, and the unstructured text data can be converted into structured data that is easy to process. For example, the first text recognition data can be segmented to obtain the phrases that constitute the sentence. Here, there are many segmentation algorithms that can be used. For example, in some embodiments, a dictionary-based segmentation algorithm can be used to first divide the first text recognition data into words according to the dictionary, and then find the best combination of words; in some embodiments, a word-based segmentation algorithm can also be used to first divide the sentence into individual words, and then combine the words into words to find the best combination. After the first text recognition data is segmented, the word embedding vector corresponding to each word in the phrase can be determined by a pre-established dictionary. Of course, in some embodiments, the word embedding vector can be obtained by mapping the word to a vector space with a unified lower dimension. The strategies for generating such mapping include neural networks, dimensionality reduction of word co-occurrence matrices, probability models, and interpretable knowledge base methods. Taking the word embedding vector as the structured data obtained by encoding the word as an example, after obtaining the word embedding vector corresponding to each word in the first text recognition data, these word embedding vectors can be used as the fourth feature information. Of course, these word embedding vectors can also be accumulated, and the accumulated vectors can be recorded as phrase vectors. The phrase vectors are normalized to obtain the vector corresponding to the first text recognition data, and the vector is used as the fourth feature information. This application does not impose specific restrictions on this.

[0170] by Figure 8 Taking the model structure shown in as an example, when the text adaptation sub-model is used to perform domain adaptation processing of text content, for example, the input data is initial text data (s1, s2...s n, n is a positive integer), and after inputting it into the text encoder, the fourth feature information obtained by the text encoder includes multiple fixed-length vectors. These fixed-length vectors are sequentially passed to the second text decoder, which uses the vector as the hidden layer input and a start flag as the input of the current position; the output vectors (such as p1, p2...p j , j is a positive integer) corresponds to a unit of text data (generally corresponding to a word). In addition, the second text encoder will pass the output vector of the mapping output to the next unit node, thereby completing the sequence output of the entire target text data. Based on the output vectors p1, p2...p j The corresponding target text data can be obtained through dictionary translation and other methods.

[0171] In the embodiment of the present application, after completing the first stage of training of the second speech recognition model, the second stage of training may be performed on it. The training process of the second stage may include:

[0172] Inputting the first speech sample data and the first text recognition data into an initialized second speech recognition model for speech recognition processing, and obtaining second text recognition data output by the second speech recognition model;

[0173] Determine a first loss value for training according to the second text recognition data and the first text label data;

[0174] According to the first loss value, the parameters of the second speech recognition model are updated to obtain a trained second speech recognition model.

[0175] In the embodiment of the present application, when the second speech recognition model is trained in the second stage, the training data obtained above can still be used, that is, the first speech sample data of the first field, the first text label data of the first speech sample data, and the first text recognition data of the first speech sample data. Of course, in some embodiments, after obtaining these training data, the data can be divided into two parts, one part of which is used for the first stage of training and the other part is used for the second stage of training. The specific data division ratio of this application is not limited.

[0176] For the second speech recognition model after initialization training, the first speech sample data and the first text recognition data can then be input into the initialized second speech recognition model for speech recognition processing. The specific processing process is similar to the aforementioned step 304 and will not be described in detail here. In an embodiment of the present application, the text data output by the obtained second speech recognition model is recorded as the second text recognition data. Similarly, based on the second text recognition data and the first text label data, the accuracy of the second speech recognition model in recognizing the first speech sample data can be determined. Specifically, the deviation between the second text recognition data and the first text label data can be determined to obtain the loss value corresponding to the second speech recognition model, which is recorded as the first loss value. After obtaining the first loss value, the accuracy of the second speech recognition model in speech recognition can be evaluated according to the size of the first loss value, so as to perform back-propagation training on the second speech recognition model and update the relevant parameters inside it. The specific training process is similar to the aforementioned speech adaptation sub-model and will not be described in detail here.

[0177] It should be noted that in the embodiment of the present application, when training the second speech recognition model, the first text recognition data obtained by performing speech recognition processing on the first speech sample data by the first speech recognition model needs to be used. Therefore, before training the second speech recognition model, the first speech recognition model needs to be trained, and the specific process of training the first speech recognition model may include the following steps:

[0178] Acquire second speech sample data of the second domain and second text label data of the second speech sample data; the second text label data is used to represent the real text content of the second speech sample data;

[0179] The first speech recognition model is trained using the second speech sample data and the second text label data to obtain a trained first speech recognition model.

[0180] In an embodiment of the present application, when training the first speech recognition model, second speech sample data of the second field and second text label data of the second speech sample data can be obtained. Here, the second field is used to characterize the field scope of speech data, which is different from the aforementioned first field, and can be a data range of any smaller field, or a data range of a larger field including speech data of various target objects. Of course, in order to improve the generalization performance of the first speech recognition model as much as possible, so that the accuracy of the initial text data obtained is at a higher level, in an embodiment of the present application, the data range contained in the second field can be set to be larger. For example, it can cover multiple languages ​​and involve speech data of people of various ages.

[0181] The speech data in the second field is recorded as the second speech sample data. In the embodiment of the present application, the second text label data is also obtained synchronously, which is used to characterize the real text content of the second speech sample data. After obtaining the above-mentioned training data, the second speech sample data can be input into the first speech recognition model for speech recognition processing to obtain the corresponding initial text data, and then the parameters of the first speech recognition model are updated according to the initial text data and the second text label data, so as to obtain the trained first speech recognition model. The specific training process can be similar to the aforementioned speech adaptation sub-model, and this application will not go into details here.

[0182] It should be noted that in the embodiment of the present application, the model structure of the speech adaptation submodel and the first speech recognition model can be the same. Both can be trained in the form of transfer learning. Specifically, when building the entire speech recognition system, the first speech recognition model can be trained first, the second domain corresponding to the first speech recognition model is the source domain, and the first domain corresponding to the speech adaptation submodel is the target domain. Based on transfer learning, the target domain task can use the domain information (domain-specificinformation) possessed by the training data of the source domain task (Source task) to improve its generalization performance and accuracy, and help each other learn by sharing the underlying feature representation (sharedrepresentation), thereby improving the model performance of multiple related tasks at the same time. Specifically in the application of the present application, that is, after the first speech recognition model is trained, the initialized speech adaptation submodel can be constructed according to the model structure and model parameters of the first speech recognition model. Then, a small amount of training data of the first domain is used to fine-tune the initialized speech adaptation submodel, so that parameter convergence can be quickly achieved, which greatly speeds up the training speed of the model and reduces the consumption of computing resources.

[0183] Specifically, in a possible implementation, after the step of acquiring the first text label data and the first text recognition data, the method further includes:

[0184] Randomly selecting original text data from the first text label data or the first text recognition data;

[0185] Transform the original text data to construct virtual text recognition data;

[0186] The first text recognition data is augmented by the virtual text recognition data.

[0187] In the embodiment of the present application, the function of the text adaptation sub-model of the second speech recognition model is to perform domain adaptation in text modality on the initial text data recognized by the first speech recognition model. In other words, the text adaptation sub-model needs to map the text data containing errors recognized in the original domain to text data without errors in the target domain as much as possible. Therefore, in the embodiment of the present application, in order to improve the domain adaptation ability of the text adaptation sub-model, the first text recognition data can be amplified, that is, some virtual first text recognition data that may appear in the first speech recognition model can be added.

[0188] Specifically, when the first text recognition data is amplified, the first text label data or the first text recognition data may be transformed to obtain virtual text recognition data, and the first text recognition data may be expanded by the virtual text recognition data. Here, in order to reduce the amount of data processing, a portion of data may be randomly selected from the first text label data or the first text recognition data, recorded as original text data, and then the original text data may be transformed to obtain virtual text recognition data.

[0189] Specifically, in a possible implementation, the original text data is transformed to construct virtual text recognition data, including:

[0190] Perform word segmentation processing on the original text data to obtain original word segmentation phrases corresponding to the original text data; the original word segmentation phrases include a number of words;

[0191] Transform the original segmented word phrase to obtain a transformed target segmented word phrase;

[0192] According to the target segmented word phrase, virtual text recognition data is constructed.

[0193] In the embodiment of the present application, when the original text data is transformed to obtain the virtual text recognition data, the original text data can be segmented. Similarly, there are many segmentation algorithms that can be used in the embodiment of the present application. For example, in some embodiments, a dictionary-based segmentation algorithm can be used to first divide the original text data into words according to the dictionary, and then find the best combination of words; in some embodiments, a word-based segmentation algorithm can also be used to first divide the original text data into individual words, and then combine the words into words to find the best combination. After the original text data is segmented, the corresponding original segmentation phrase can be obtained, which includes several words. Then, the original segmentation phrase can be transformed, for example, some (at least one) words can be deleted or replaced, or preset words can be inserted between two adjacent words. Of course, it can be understood that the specific implementation of the above-mentioned transformation processing can be set according to demand, and this application does not limit this.

[0194] Specifically, in a possible implementation, performing replacement processing on at least one word includes:

[0195] Extracting first audio feature information of the word to be replaced;

[0196] Perform a matching search in the dictionary to determine a target word; wherein the similarity between the second audio feature information and the first audio feature information of the target word is higher than a preset threshold;

[0197] The target word is used to replace the word to be replaced.

[0198] In an embodiment of the present application, a method for replacing words in an original word segmentation phrase is provided. Specifically, first, the words to be replaced can be determined from the original word segmentation words. The words can be randomly selected or manually selected, and the present application does not limit this. After the words to be replaced are determined, the audio feature information of the words can be extracted and recorded as the first audio feature information. The extraction method of the first audio feature information is similar to the aforementioned third feature information, and will not be repeated here. Then, a matching search can be performed in the dictionary through the first audio feature information. Specifically, the target word can be determined by the similarity between the audio feature information of each word in the dictionary and the first audio feature information. For example, when the similarity between the audio feature information of a word and the first audio feature information is higher than a preset similarity threshold, it can be considered that the word and the word to be replaced have a strong confusion in pronunciation, and it can be determined as the target word. In the embodiment of the present application, the audio feature information of each word in the dictionary is recorded as the second audio feature information. After the target word is determined, the target word can be used to replace the word to be replaced, thereby constructing virtual text recognition data.

[0199] It can be understood that in the embodiment of the present application, by constructing virtual text recognition data to amplify the first text recognition data, it is possible to facilitate the text adaptation sub-model to more effectively perform domain adaptation on the text modality of the speech data, thereby improving the accuracy of the recognition results obtained by the second recognition model.

[0200] In order to compare and verify the effectiveness of the speech recognition method provided in the embodiment of the present application, the target text data obtained based on the speech recognition method provided in the embodiment of the present application is compared with the model recognition results under various other strategies. Fig. 9 , Fig. 9The WER data corresponding to each recognition strategy are shown. Among them, other speech recognition strategies include the domain adaptation method (LM) and ASR error correction method that only use text modality, the method of fine-tuning the ASR model using audio data and text data (ASR TL), the original ASR recognition result (E2E ASR), and a related method: first do ASR fine-tuning, and then do ASR error correction based on text. Fig. 9 It can be seen from the results shown that the recognition method provided in the embodiment of the present application has a significant improvement in the WER index compared with other previous methods.

[0201] Reference Fig.10 , conduct in-depth analysis on a specific speech recognition case, such as the speech data of "The name lili awakenthe tour of his mind", through Fig. 9 The text data obtained by using the partial recognition strategy shown is as follows Fig.10 As shown. It can be seen that for E2E ASR and ASR TL, both have errors in the recognition of "lili", which is mostly caused by the domain mismatch of the voice data, indicating that the above two strategies cannot solve errors with similar pronunciations. The ASR error correction method, on the other hand, produced a text recognition error for "tours". Based on the recognition method provided in the embodiments of the present application, accurate speech recognition results can be obtained through comprehensive domain adaptation of speech modality and text modality, which is better than several other recognition strategies.

[0202] Below, in conjunction with some more specific embodiments, the application of the speech recognition method provided in this application is described in detail.

[0203] Embodiment 1:

[0204] In the field of education, oral test is a systematic and orderly language proficiency test project, which aims to test the listening and reading proficiency of the testee in a specified type of language. Generally speaking, after scientific and objective testing, the corresponding score results of the testee can be obtained. The score results are widely considered in many scenarios such as enrollment, employment, immigration, etc., and are of great significance to the work and life of individuals. Therefore, the accuracy of the score results is very important for oral test.

[0205] The speech recognition method in the embodiment of the present application can be applied to application scenarios related to oral tests and exercises. For example, if an educational institution hopes to accurately evaluate the scores of candidates participating in the oral test, the speech recognition method provided in the embodiment of the present application can be used to recognize the candidate's voice data to obtain the text data corresponding to the voice data. Then, the examiner or the artificial intelligence system determines the candidate's score result based on the recognized text data. In other embodiments, for example, before taking a formal oral test, the candidate hopes to know his or her oral level in advance in order to better arrange learning tasks. The speech recognition method provided in the embodiment of the present application can be used to recognize his or her own voice data, and the recognized text data can be fed back, so as to facilitate the candidate to carry out targeted error correction and improvement. Of course, it should be noted that the above application scenarios only serve as an example and do not mean to limit the actual application of the method in the embodiment of the present application.

[0206] It is understandable that, since the oral test targets a relatively diverse population, including persons of different age groups and different oral levels. Therefore, in the embodiment of the present application, a general speech recognition model can be first established, i.e., the aforementioned first speech recognition model, and then multiple corresponding field adaptation models, i.e., multiple second speech recognition models, can be established for different fields. For example, in some embodiments, different fields can be divided according to the region where the personnel are located. In this case, the first speech recognition model can be obtained by training based on the speech data of personnel within any geographical range, and the geographical range can be recorded as the first geographical range. Generally speaking, the first geographical range can be set to be relatively broad. And multiple second speech recognition models can be obtained by training based on the speech data of personnel within a specific geographical range, respectively, and the specific geographical range here is recorded as the second geographical range. It is understandable that the second geographical range is the field to which the second speech recognition model is adapted, and therefore, the geographical information to which the speech data source belongs can be used as the field information to distinguish whether the speech data and the second speech recognition model are adapted.

[0207] Specifically, refer to Fig.11 , Fig.11 A business data flow diagram suitable for oral test application scenarios is shown below. Fig.11 , briefly describe the above application process.

[0208] In the embodiment of the present application, the overall system architecture can refer to Figure 2 On the candidate side, you can use your terminal device to download and install the corresponding application, refer to Fig.12 , Fig.12 The figure shows a schematic diagram of the interface of a spoken language test APP for speech recognition. Fig.12The oral test questions shown in the figure belong to the type of describing pictures. The examinee needs to observe the given image and use a piece of voice data in a specified language to describe the content contained in the image. When the voice recognition application is specifically executed, the examinee can enter the geographical information of the area where he is located in the area output box 1210. For example, the text information of "Guangdong" can be entered as the geographical information. In this way, the server can use the geographical information as the domain information and determine the corresponding domain adaptation model. Then you can click the virtual button "Start Recording" to start this oral test. At this time, the terminal device will collect and record the examinee's voice data. After completing the oral test, you can click the virtual button "Finish Recording" so that the APP can transmit the candidate's voice data and the input geographical information to the background server. After receiving the candidate's voice data, the server can send it to the speech recognition model (i.e., the first speech recognition model in this application) to obtain the initial recognition text (i.e., the initial text data in this application); then, according to the candidate's geographical information, the corresponding domain adaptation model is determined from multiple domain adaptation models (i.e., the second speech recognition model in this application), and the candidate's voice data and the initial recognition text are sent to the domain adaptation model to obtain a more accurate recognition text. Then, the server can return the recognition result to the terminal device, refer to Fig.12 After receiving the recognition result, the terminal device can display it in the text display box 1220, so as to conveniently and intuitively understand the performance of this oral test quickly and easily.

[0209] It can be understood that in the embodiments of the present application, regional information can be used as domain information, and a suitable second speech recognition model can be selected through domain adaptation of the regional information, which can better overcome the local accent problems that may exist among people from different regions and achieve more accurate recognition of speech data.

[0210] Embodiment 2:

[0211] At present, the development of mobile Internet has profoundly changed people's lifestyles. The content presented by various audio resources can give people a rich audio-visual experience, greatly satisfying everyone's entertainment and leisure needs. For example, the popular short videos often support creators to edit and produce freely, which is very flexible. However, in most cases, the short video content produced by creators is not configured with corresponding subtitles, or subtitles in a specified language cannot be provided, which brings a bad experience to viewers in different regions.

[0212] Therefore, based on the speech recognition method provided in the embodiment of the present application, the speech data in the short video data can be effectively recognized, and the corresponding text data can be determined. Then, the recognized text data can be displayed in the subtitle position below the short video, so that people who watch the short video can easily and conveniently understand the video content based on the text data, greatly improving the viewing experience of the short video; and the application of the speech recognition method can freely provide the recognition and translation functions of speech data in various languages, reducing the workload of video creators. For the application scenarios in the embodiments of the present application, for example, there is a need to convert speech data in different languages ​​into corresponding text data in a certain language. In this case, the first speech recognition model and the second speech recognition model are both used to output text data in a certain language. At this time, the field can be divided according to the language corresponding to the speech data in the short video, and the first speech recognition model can be obtained by training based on speech data in at least one language. And multiple second speech recognition models can be obtained by training based on speech data in different languages. Therefore, in the embodiment of the present application, the language information corresponding to the speech data can be used as the field information to distinguish whether the speech data and the second speech recognition model are adapted.

[0213] Of course, the speech recognition method in the embodiment of the present application is not limited to the application in the speech recognition of short videos. In some movie and TV play playback software, a speech recognition component integrated with the speech recognition method in the embodiment of the present application can also be provided. Fig.13 , Fig.13 A video playback software with a voice recognition function is shown. In the software, a corresponding voice recognition operation mark 1310 can be set on the playback interface. The voice recognition function can be turned on by clicking the voice recognition operation mark 1310, so that text data recognized based on the voice data of the video can be displayed in a subtitle box 1320 below the video.

[0214] It should be supplemented that, for the above-mentioned speech recognition method applied in the video field, the domain information of the speech data can be determined according to the region of the video publisher or the language of the speech data in the video, but is not limited thereto.

[0215] It is understandable that the speech recognition method of the embodiment of the present application is implemented in the above-mentioned short video application and video playback software, and the overall architecture of the system can be set with reference to the first embodiment, and this application will not elaborate on this. Fig.14 , Fig.14 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application, the speech recognition device comprising:

[0216] An acquisition module 1410 is used to acquire speech data of a target object and domain information of the speech data;

[0217] The first recognition module 1420 is used to input the speech data into the first speech recognition model for speech recognition processing to obtain initial text data;

[0218] An adaptation module 1430 is used to determine a second speech recognition model adapted to the domain of the speech data according to the domain information;

[0219] The second recognition module 1440 is used to extract first feature information of speech data and second feature information of initial text data through a second speech recognition model, and perform speech recognition processing based on the first feature information and the second feature information to obtain target text data.

[0220] Further, the first speech recognition model includes an audio encoder and a first text decoder; the first recognition module includes:

[0221] An audio encoding submodule, used for inputting the speech data into the first speech recognition model, encoding the speech data through the audio encoder, and obtaining third feature information of the speech data;

[0222] The first text decoding submodule is used to decode the third feature information through a first text decoder to obtain initial text data.

[0223] Furthermore, the second speech recognition model includes a speech adaptation sub-model and a text adaptation sub-model; the second recognition module includes:

[0224] A first feature extraction submodule, used to extract first feature information of speech data through a speech adaptation submodel;

[0225] A second feature extraction submodule, used to extract second feature information of the initial text data through a text adaptation submodel;

[0226] A feature fusion submodule, used to fuse the first feature information and the second feature information to obtain fused feature information;

[0227] The recognition submodule is used to obtain the target text data based on the fused feature information.

[0228] Furthermore, the device also includes a first training module, and the first training module includes:

[0229] A first acquisition submodule is used to acquire first speech sample data of a first domain, first text label data of the first speech sample data, and first text recognition data of the first speech sample data; the first text label data is used to characterize the real text content of the first speech sample data, and the first text recognition data is obtained by performing speech recognition processing on the first speech sample data by using a trained first speech recognition model;

[0230] A first processing submodule, used for inputting the first speech sample data and the first text recognition data into the initialized second speech recognition model for speech recognition processing, and obtaining the second text recognition data output by the second speech recognition model;

[0231] A first loss value calculation submodule, used to determine a first loss value of training according to the second text recognition data and the first text label data;

[0232] The first updating submodule is used to update the parameters of the second speech recognition model according to the first loss value to obtain a trained second speech recognition model.

[0233] Furthermore, the device also includes an amplification module, which includes:

[0234] A selection submodule, used for randomly selecting original text data from the first text label data or the first text recognition data;

[0235] The transformation submodule is used to transform the original text data to construct virtual text recognition data;

[0236] The data adding submodule is used to amplify the first text recognition data by using virtual text recognition data.

[0237] Furthermore, the above transformation submodule is specifically used for:

[0238] Perform word segmentation processing on the original text data to obtain original word segmentation phrases corresponding to the original text data; the original word segmentation phrases include a number of words;

[0239] Transform the original segmented word phrase to obtain a transformed target segmented word phrase;

[0240] According to the target segmented word phrase, virtual text recognition data is constructed.

[0241] Further, the above-mentioned transformation submodule is specifically used to perform at least one of the following steps:

[0242] Deleting at least one word;

[0243] Alternatively, at least one word is replaced;

[0244] Alternatively, a preset word is inserted between at least two adjacent words.

[0245] Furthermore, the above transformation submodule is specifically used for:

[0246] Extracting first audio feature information of the word to be replaced;

[0247] Perform a matching search in the dictionary to determine a target word; wherein the similarity between the second audio feature information and the first audio feature information of the target word is higher than a preset threshold;

[0248] The target word is used to replace the word to be replaced.

[0249] Furthermore, the device also includes a second training module, and the second training module includes:

[0250] A second processing submodule is used to input the first speech sample data into the speech adaptation submodel for speech recognition processing to obtain third text recognition data;

[0251] A second loss value calculation submodule, used to determine a second loss value of training according to the third text recognition data and the first text label data;

[0252] A second updating submodule, used to update the parameters of the speech adaptation submodel according to the second loss value to obtain a trained speech adaptation submodel;

[0253] A third processing submodule is used to input the first text recognition data into the text adaptation submodel to perform domain adaptation processing on the text content to obtain fourth text recognition data;

[0254] A third loss value calculation submodule, used to determine a third loss value of training according to the fourth text recognition data and the first text label data;

[0255] A third updating submodule, used for updating the parameters of the text adaptation submodel according to the third loss value to obtain a trained text adaptation submodel;

[0256] The initialization submodule is used to obtain an initialized second speech recognition model based on the trained speech adaptation submodel and the trained text adaptation submodel.

[0257] Furthermore, the text adaptation sub-model includes a text encoder and a second text decoder; the third processing sub-module is specifically used for:

[0258] Inputting the first text recognition data into the text adaptation sub-model, encoding the first text recognition data through a text encoder, and obtaining fourth feature information of the first text recognition data;

[0259] The fourth feature information is decoded by the second text decoder to obtain fourth text recognition data.

[0260] Furthermore, the device also includes:

[0261] A second acquisition module is used to acquire second voice sample data of a second field and second text label data of the second voice sample data; the second text label data is used to represent the real text content of the second voice sample data;

[0262] A third training module is used to train the first speech recognition model using the second speech sample data and the second text label data to obtain a trained first speech recognition model;

[0263] The construction module is used to construct an initialized speech adaptation sub-model according to the model structure and model parameters of the trained first speech recognition model.

[0264] It is understandable that if Figure 3 The contents of the speech recognition method embodiment shown in the figure are applicable to the speech recognition device embodiment. The functions specifically implemented by the speech recognition device embodiment are similar to those in the embodiment shown in the figure. Figure 3 The speech recognition method embodiment shown is the same as that shown in FIG. Figure 3 The beneficial effects achieved by the illustrated speech recognition method embodiment are also the same.

[0265] Reference Fig.15 , the embodiment of the present application further discloses an electronic device, including:

[0266] at least one processor 1510;

[0267] At least one memory 1520, used to store at least one program;

[0268] When at least one program is executed by at least one processor 1510, at least one processor 1510 implements the following Figure 3 An embodiment of a speech recognition method is shown.

[0269] It is understandable that if Figure 3 The contents of the speech recognition method embodiment shown in the figure are applicable to the electronic device embodiment. The functions specifically implemented by the electronic device embodiment are similar to those in the embodiment shown in the figure. Figure 3 The speech recognition method embodiment shown is the same as that shown in FIG. Figure 3 The beneficial effects achieved by the illustrated speech recognition method embodiment are also the same.

[0270] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the following Figure 3 An embodiment of a speech recognition method is shown.

[0271] Understandably, Figure 3The contents of the speech recognition method embodiment shown in the figure are applicable to the computer-readable storage medium embodiment. The functions specifically implemented by the computer-readable storage medium embodiment are similar to those of Figure 3 The speech recognition method embodiment shown is the same as that of Figure 3 The beneficial effects achieved by the illustrated speech recognition method embodiment are also the same.

[0272] The embodiment of the present application further discloses a computer program product or a computer program, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in the above-mentioned computer-readable storage medium; Fig.15 The processor of the electronic device shown can read the computer instruction from the above-mentioned computer readable storage medium, and the processor executes the computer instruction so that the electronic device executes Figure 3 An embodiment of a speech recognition method is shown.

[0273] Understandably, Figure 3 The contents of the speech recognition method embodiment shown in the figure are all applicable to the computer program product or computer program embodiment, and the functions specifically implemented by the computer program product or computer program embodiment are the same as those in the embodiment of the present invention. Figure 3 The speech recognition method embodiment shown is the same as that of Figure 3 The beneficial effects achieved by the illustrated speech recognition method embodiment are also the same.

[0274] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0275] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0276] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0277] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, apparatus, or device and execute instructions), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, a "computer-readable storage medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0278] More specific examples of computer-readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or processing in another suitable manner as necessary, and then stored in a computer memory.

[0279] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0280] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0281] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0282] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A speech recognition method, characterized in that: The method comprises: Acquire voice data of a target object and domain information of the voice data; Inputting the speech data into a first speech recognition model for speech recognition processing to obtain initial text data; Determining, according to the domain information, a second speech recognition model adapted to the domain of the speech data; The first feature information of the speech data and the second feature information of the initial text data are extracted by the second speech recognition model, and speech recognition processing is performed based on the first feature information and the second feature information to obtain target text data.

2. The speech recognition method according to claim 1, characterized in that: The first speech recognition model includes an audio encoder and a first text decoder; the step of inputting the speech data into the first speech recognition model for speech recognition processing to obtain initial text data includes: Inputting the speech data into the first speech recognition model, encoding the speech data through the audio encoder, and obtaining third feature information of the speech data; The third characteristic information is decoded by the first text decoder to obtain the initial text data.

3. The speech recognition method according to claim 1, characterized in that: The second speech recognition model includes a speech adaptation sub-model and a text adaptation sub-model; the extracting the first feature information of the speech data and the second feature information of the initial text data by the second speech recognition model, and performing speech recognition processing based on the first feature information and the second feature information to obtain the target text data includes: Extracting first feature information of the speech data by using the speech adaptation sub-model; Extracting second feature information of the initial text data by using the text adaptation sub-model; fusing the first feature information and the second feature information to obtain fused feature information; The target text data is obtained according to the fused feature information.

4. The speech recognition method according to claim 3, characterized in that: The second speech recognition model is trained by the following steps: Acquire first speech sample data of a first domain, first text label data of the first speech sample data, and first text recognition data of the first speech sample data; the first text label data is used to characterize the real text content of the first speech sample data, and the first text recognition data is obtained by performing speech recognition processing on the first speech sample data by using a trained first speech recognition model; Inputting the first speech sample data and the first text recognition data into an initialized second speech recognition model for speech recognition processing, and obtaining second text recognition data output by the second speech recognition model; Determine a first loss value for training according to the second text recognition data and the first text label data; According to the first loss value, the parameters of the second speech recognition model are updated to obtain a trained second speech recognition model.

5. The speech recognition method according to claim 4, characterized in that: After the step of acquiring the first text label data and the first text recognition data, the method further includes: Randomly selecting original text data from the first text tag data or the first text recognition data; Transforming the original text data to construct virtual text recognition data; The first text recognition data is augmented by the virtual text recognition data.

6. The speech recognition method according to claim 5, characterized in that: The step of transforming the original text data to construct virtual text recognition data includes: Performing word segmentation processing on the original text data to obtain an original word segmentation phrase corresponding to the original text data; the original word segmentation phrase includes a plurality of words; Transforming the original segmented word phrase to obtain a transformed target segmented word phrase; The virtual text recognition data is constructed based on the target segmented word phrase.

7. The speech recognition method according to claim 6, characterized in that: The transforming of the original segmented word phrase comprises at least one of the following steps: Deleting at least one of the words; Alternatively, at least one of the words is replaced; Alternatively, a preset word is inserted between at least two adjacent words.

8. The speech recognition method according to claim 7, characterized in that: The replacing process of at least one of the words comprises: Extracting first audio feature information of the word to be replaced; Perform a matching search in the dictionary to determine a target word; wherein the similarity between the second audio feature information of the target word and the first audio feature information is higher than a preset threshold; The target word is used to replace the word to be replaced.

9. The speech recognition method according to claim 4, characterized in that: The second speech recognition model is also initialized and trained by the following steps: Inputting the first speech sample data into the speech adaptation sub-model for speech recognition processing to obtain third text recognition data; Determining a second loss value for training according to the third text recognition data and the first text label data; According to the second loss value, updating the parameters of the speech adaptation sub-model to obtain a trained speech adaptation sub-model; Inputting the first text recognition data into the text adaptation sub-model to perform domain adaptation processing on the text content to obtain fourth text recognition data; Determining a third loss value for training according to the fourth text recognition data and the first text label data; According to the third loss value, updating the parameters of the text adaptation sub-model to obtain a trained text adaptation sub-model; Based on the trained speech adaptation sub-model and the trained text adaptation sub-model, an initialized second speech recognition model is obtained.

10. The speech recognition method according to claim 9, characterized in that: The text adaptation sub-model includes a text encoder and a second text decoder; the first text recognition data is input into the text adaptation sub-model to perform domain adaptation processing of the text content to obtain fourth text recognition data, including: Inputting the first text recognition data into the text adaptation sub-model, encoding the first text recognition data through the text encoder, and obtaining fourth feature information of the first text recognition data; The fourth feature information is decoded by the second text decoder to obtain the fourth text recognition data.

11. The speech recognition method according to claim 3, characterized in that: The method further comprises: Acquire second speech sample data of a second domain and second text label data of the second speech sample data; the second text label data is used to represent the real text content of the second speech sample data; Training the first speech recognition model using the second speech sample data and the second text label data to obtain a trained first speech recognition model; An initialized speech adaptation sub-model is constructed according to the model structure and model parameters of the trained first speech recognition model.

12. A speech recognition device, characterized in that: The device comprises: An acquisition module, used to acquire the voice data of a target object and the domain information of the voice data; A first recognition module, used for inputting the speech data into a first speech recognition model for speech recognition processing to obtain initial text data; An adaptation module, configured to determine, based on the domain information, a second speech recognition model adapted to the domain of the speech data; The second recognition module is used to extract the first feature information of the speech data and the second feature information of the initial text data through the second speech recognition model, and perform speech recognition processing based on the first feature information and the second feature information to obtain target text data.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the speech recognition method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the speech recognition method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the speech recognition method according to any one of claims 1 to 11 is implemented.

Images