Speech recognition model training method, speech recognition method and related device

Abstract

The present application relates to the field of artificial intelligence technology, and provides a training method for a speech recognition model, a speech recognition method and related devices, to solve the problem of low accuracy when the speech recognition model performs speech recognition. Before inputting the audio feature data of the audio file into the speech recognition model, the present application masks the audio feature data with the model receptive field as the interval, and then inputs the masked audio feature data into the speech recognition model, and trains the speech recognition model based on the audio feature data. Due to the masking process of the present application, the samples include a variety of samples such as with context, without context, without previous context, and without next context, so that the speech recognition model can learn the recognition results under different circumstances. In summary, the present application can get rid of the excessive dependence of the speech recognition model on partial information by masking the audio feature data with equal receptive field intervals, and improve the recognition rate of the speech recognition model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method based on a speech recognition model, a speech recognition method and related devices. Background Art

[0002] With the development and upgrading of speech recognition technology, people have higher and higher requirements on the recognition performance of speech recognition models.

[0003] At present, speech recognition technology mainly relies on deep learning technology, that is, the speech frame features are input into the neural network model, the neural network obtains the predicted phoneme probability value of each frame, and then decodes it through the decoder to obtain the final speech sequence. In this process, the accuracy of speech recognition of the trained model is low. Summary of the invention

[0004] The purpose of this application is to provide a training method for a speech recognition model, a speech recognition method and related devices, so as to solve the problem of low accuracy when the speech recognition model performs speech recognition.

[0005] In a first aspect, the present application provides a method for training a speech recognition model, the method comprising:

[0006] Inputting audio feature data of the first audio file into the speech recognition model; wherein the audio feature data is obtained by masking processing, and the masking processing is used to mask the audio feature data at intervals of the receptive field;

[0007] Based on the audio feature data, a speech recognition model is trained.

[0008] In some embodiments, masking is performed on the audio feature data, and the method includes:

[0009] determining at least two of the following masking parameters: a masking start point, a masking length, and a masking end point;

[0010] The audio feature data within the data range indicated by the masking parameter is replaced.

[0011] In some embodiments, the method of replacing the audio feature data within the data range indicated by the masking parameter further comprises:

[0012] The masking parameter indicates that the audio feature data within the data range is replaced with a specified value or a random number.

[0013] In some embodiments, the method of filtering out data that needs masking processing from the audio feature data includes:

[0014] The audio feature data are sampled at equal intervals, and the size of each sampling area is the same as the receptive field size.

[0015] In some embodiments, the sampling interval is the receptive field size.

[0016] In some embodiments, the masking length is less than or equal to 1 / n of the receptive field, where n is a positive integer.

[0017] In some embodiments, the method further comprises:

[0018] Filtering out the first audio feature file from the training sample set, and using the remaining audio files as the second audio files;

[0019] The training samples used to train the speech recognition model also include audio feature data of the second audio file.

[0020] In a second aspect, the present application further provides a speech recognition method, the method comprising:

[0021] Get the speech to be recognized;

[0022] The speech to be recognized is input into the speech recognition model as described in any one of the first aspects to obtain a speech recognition result.

[0023] In a third aspect, the present application further provides an electronic device, including:

[0024] A display, a user interface for displaying audio file data;

[0025] A memory for storing executable instructions of the processor;

[0026] A processor is used to execute the executable instructions to implement any method provided in the first aspect or the second aspect of the present application.

[0027] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any method provided in the first aspect or the second aspect of the present application.

[0028] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any method provided in the first aspect or the second aspect of the present application.

[0029] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0030] Based on the training method of the above-mentioned speech recognition model, the robustness of the speech recognition model is enhanced. When users use the speech recognition model for speech recognition, the model's excessive dependence on certain information is reduced, so that the model can better balance current information and contextual information, thereby improving the recognition rate of the speech recognition model and improving the user experience.

[0031] It should be understood that the above general description and the following detailed description are only exemplary and explanatory and cannot limit the present application. Based on the common sense in the art, the above preferred conditions can be combined arbitrarily to obtain the preferred embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a method for training a speech recognition model provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a sample preparation process for a method for training a speech recognition model provided in an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the receptive field provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of the effect of performing equal receptive field interval masking on audio feature data provided in an embodiment of the present application;

[0038] Figure 6 Another flowchart of a method for training a speech recognition model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] Furthermore, in the description of the embodiments of the present application, unless otherwise specified, “ / ” means or. For example, A / B can mean A or B. The “and / or” in the text is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “plurality” refers to two or more than two.

[0041] In the following, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more.

[0042] In order to solve the problem of low accuracy of speech recognition model during speech recognition, the inventor has studied and analyzed the working principle of speech recognition model. The research and analysis show that in order to improve the recognition performance of speech recognition model, the current speech recognition technology recognizes that the range recognized by the phoneme output network for each predicted frame is not just the audio frame feature input of the current frame, but the continuous audio frame features within a certain range. This range is called the receptive field of the neural network. In addition to the input frame corresponding to the output frame, the remaining frames in the receptive field are called context information. The model fits the data in the receptive field and obtains the predicted value of the current frame based on the data in the receptive field. However, due to problems such as data imbalance, the speech recognition model will have "training bias", which is manifested in excessive reliance on context information and excessive reliance on current input information. The semantic information brought by the context cannot be learned, resulting in redundant parameters. In view of this, the embodiment of the present application provides a training method, device, storage medium and program product for a speech recognition model.

[0043] The following will introduce the training method of the speech recognition model provided by this application in conjunction with the embodiments.

[0044] The inventive concept of the present application can be summarized as follows: before the audio feature data of the audio file is input into the speech recognition model, the audio feature data is masked with the receptive field as the interval, and then the masked audio feature data is input into the speech recognition model, and the speech recognition model is trained based on the audio feature data. Since the masking process with equal receptive field intervals is performed, the samples include a variety of samples such as with context, without context, without previous context, and without next context, so that the speech recognition model can learn the recognition results under different circumstances, thereby improving the robustness of the model. In summary, the embodiments of the present application can reduce the excessive dependence of the speech recognition model on some information by performing masking process with equal receptive field intervals on the audio feature data, thereby improving the recognition rate of the speech recognition model and improving the user experience.

[0045] After introducing the main inventive ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0046] first, Figure 1 A schematic structural diagram of an electronic device 100 is shown.

[0047] Figure 1 exemplarily shows a hardware configuration block diagram of the electronic device 100 according to an exemplary embodiment. The embodiment is specifically described below taking the electronic device 100 as an example. It should be understood that Figure 1 The electronic device 100 shown is only an example, and the electronic device 100 may have more Figure 1 The more or less components shown in the figure can be combined with two or more components, or can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0048] Figure 1 This is a schematic diagram of the structure of an electronic device provided by the present application. Based on the above embodiments, the present application also provides an electronic device, such as Figure 1 As shown, it includes: a processor 101, a communication interface 102, an audio circuit 103, a display 104, a memory 105 and a communication bus 106, wherein the processor 101, the communication interface 102, the audio circuit 103, the display 104 and the memory 105 communicate with each other through the communication bus 106;

[0049] The memory 105 can be used to store software programs and data. The processor 101 executes various functions and data processing of the electronic device 100 by running the software programs or data stored in the memory 105. The memory 105 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 105 stores an operating system that enables the electronic device 100 to run. In the present application, the memory 105 can store an operating system and various application programs, and can also store program codes for executing the methods described in the embodiments of the present application.

[0050] The display 104 can be used to receive input digital or character information and generate signal input related to user settings and function control of the electronic device 100. Specifically, the display 104 can be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the electronic device 100. Specifically, the display 104 can include a display screen arranged on the front of the electronic device 100. The display screen can be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit can be used to display the user interface of home appliances such as air conditioners in this application, such as temperature, humidity, etc.

[0051] The audio circuit 103, the speaker 1031, and the microphone 1032 can provide an audio interface between the user and the electronic device 100. The audio circuit 103 can transmit the electrical signal converted from the received audio data to the speaker 1031, which is converted into a sound signal for output. The electronic device 100 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1032 converts the collected sound signal into an electrical signal, which is received by the audio circuit 103 and converted into audio data, and then the audio data is sent to, for example, another electronic device, or the audio data is output to the memory 105 for further processing. In this application, the microphone 1032 can obtain the user's voice.

[0052] The processor 101 is the control center of the electronic device 100. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 100 and processes data by running or executing software programs stored in the memory 105 and calling data stored in the memory 105. In some embodiments, the processor 101 may include one or more processing units; the processor 101 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 101. In the present application, the processor 101 can run the operating system, application programs, user interface display and touch response, as well as the training method of the speech recognition model described in the embodiment of the present application. In addition, the processor 101 is coupled to the display 104.

[0053] The communication bus 106 mentioned in the above electronic device can be a peripheral component interconnect standard (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor. The above processor may be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0054] The electronic device 100 in the embodiment of the present application may be a speaker, a mobile phone, a tablet computer, a wearable device, a laptop computer, a television, an air conditioner, a refrigerator, a server, or other device that supports training a speech recognition model.

[0055] In the embodiments of the present application, masking processing is performed on audio feature data at intervals of receptive fields to reduce the over-dependence of the speech recognition model on partial information, thereby improving the recognition rate of the speech recognition model. For example, the adjustable temperature range of an air conditioner is from twenty degrees to thirty-one degrees. The results recognized by the air conditioner speech recognition model are "sil twenty degrees", "sil twenty-one degrees", "sil twenty-two degrees", "sil twenty-three degrees", "sil twenty-four degrees", "sil twenty-five degrees", "sil twenty-six degrees", "sil twenty-seven degrees", "sil twenty-eight degrees", "sil twenty-nine degrees", "sil thirty degrees", "sil thirty-one degrees", where sil refers to the input of a silent frame. If the receptive field size of the speech recognition model is large enough to contain three characters, then when recognizing "two", the model will recognize "sil twenty". Since there is too much "sil twenty" in the training data, during the recognition of "two" in training, there will be an over-dependence on the previous "sil" and the following "ten", forming a bound relationship. During testing, since the context of "three" is the same as that of "two", it will be misrecognized as "two". Similarly, for the test example "sil two hundred", due to the different context information, "two" cannot be correctly recognized either. For another situation, during the training of a general speech recognition model, due to the huge amount of data and rich context information, the model will overly focus on the current input data and fail to learn the context information. For example, for "twenty-one", the present application hopes that the model can also achieve correct recognition with the assistance of the context when the pronunciation of "ten" is blurred.

[0056] Based on the above content, the flowchart of a training method for a speech recognition model provided by an embodiment of the present application is as Figure 2 shown and can be implemented as:

[0057] In step 201, the audio feature data of the first audio file is input into the speech recognition model.

[0058] In step 202, the speech recognition model is trained based on the audio feature data of the first audio file.

[0059] In some embodiments, in the training of the speech recognition model, it is not necessary to perform the masking processing of the present application on all data, and a part of the data can be retained as the original audio data. For example, the audio feature data of the first audio file is obtained through masking processing, while the audio feature data of the second audio file is not processed by masking and is used as the original audio data. When training the speech recognition model, the audio feature data of the first audio file and the audio feature data of the second audio file can be input into the model for training. These two types of data provide diverse samples, thereby improving the robustness and recognition rate of the audio recognition model.

[0060] In some embodiments, a flow chart of a sample preparation method of a speech recognition model training method provided in an embodiment of the present application is as follows: Figure 3 As shown, including the following:

[0061] In step 301, before the audio feature data of the first audio file is input into the speech recognition model, data requiring masking processing is screened out from the audio feature data.

[0062] In some embodiments, filtering out the data that needs masking from the audio feature data can be implemented by: determining the amount of audio feature data of the first audio file that is sampled at equal intervals, and then sampling the audio feature data at equal receptive field intervals so that the size of each sampling area is the same as the receptive field size. Figure 4 As shown, Figure 4 It means that the number of input nodes sensed by an output node is 5, and the receptive field of the output node is equal to 5, corresponding to the phoneme prediction value of input node "3". The sampling interval is the receptive field size, that is, the interval between each mask is the size of the receptive field.

[0063] In step 302, the audio feature data is masked with the receptive field as the interval, which can be implemented as follows: at least two of the following masking parameters are randomly determined with the receptive field as the interval: masking start point, masking length, and masking end point. Determining at least two of the above masking parameters determines the range of masking, wherein the masking length is less than or equal to 1 / n of the receptive field, and n is a positive integer. For example, if n is 3, then the masking length is less than or equal to 1 / 3 of the receptive field. Figure 5 In the figure, a speech signal is masked with the receptive field as the interval. The masking operation is performed in the gray vertical rectangular area in the figure. At the same time, the masking length is equal to 1 / 3 of the receptive field (the specific length is not limited). The masked interval Filed is the size of the receptive field. Therefore, during the model training process, when predicting "1", only the context information of the input data can be recognized, and only recognition can be performed based on the context information. When predicting "2", the complete information can be recognized, and recognition can be performed based on the complete information. When predicting "3", recognition is based on the complete previous information and part of the current information. When predicting "4", recognition is based on the complete following information and part of the current information. The position and length of the mask are dynamically random during training, thereby preventing the speech recognition model from over-relying on part of the information and enhancing the adaptability of the speech recognition model to global information.

[0064] In some embodiments, masking can be implemented by replacing the audio feature data within the data range indicated by the masking parameter. The replacement can be a specified value or a random number, or any value within a specified value range. For example, the audio feature data within the data range indicated by the masking parameter is replaced with 0, or the audio feature data within the data range indicated by the masking parameter is replaced with a random number of a Gaussian distribution. After the audio feature data within the data range indicated by the masking parameter is replaced, the speech recognition model will no longer obtain the real information within the masking range.

[0065] In step 303, a speech recognition model is trained based on the masked audio feature data.

[0066] In some embodiments, another flowchart of a sample preparation of a method for training a speech recognition model provided in an embodiment of the present application is as follows: Figure 6 As shown, including the following:

[0067] In step 601, an audio file sample x is obtained, and the starting point of masking is determined to be a random value within the range of 0 to the receptive field, that is, m_start=random(0, filed), where m_start represents the starting point of masking, filed represents the receptive field, and random represents the random range.

[0068] In step 602, the length of the mask is determined to be randomly 0 to 1 / 3 of the receptive field, that is, m_len=random(0, filed / 3), m_end=m_start+m_len, where m_len represents the length of the mask and m_end represents the end point of the mask.

[0069] In step 603, the audio feature data within the data range indicated by the masking parameter is replaced with 0, that is, x[m_start:m_end] = 0. Here, x[m_start:m_end] represents the masking data.

[0070] In step 604, masking processing is performed on the audio file sample x at intervals of a fixed receptive field length, ie, m_start=m_end+filed.

[0071] In step 605, it is determined whether the end of the audio file x has been reached. If the end of the audio file x has been reached, the masking process of the audio file x has been completed. If the end of the audio file x has not been reached, the process returns to step 602 to continue the masking process of the audio file x.

[0072] The above-mentioned method of masking the audio feature data with the receptive field as the interval can reduce the excessive dependence of the speech recognition model on some information within the receptive field, thereby enhancing the adaptability of the speech recognition model to global information and improving the recognition rate of the speech recognition model.

[0073] Based on the same inventive concept, the present application also provides a speech recognition method, in which the speech to be recognized can be obtained and then input into the trained speech recognition model described above, thereby obtaining a speech recognition result. Afterwards, the electronic device can perform corresponding operations based on the speech recognition result. For example, the air conditioner can recognize that the user's instruction is to adjust the temperature to 21°C, and the air conditioner will cool or heat according to the set target temperature of 21°C.

[0074] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0075] Similar parts between the embodiments provided in this application can be referenced to each other. The specific implementation methods provided above are only a few examples under the general concept of this application and do not constitute a limitation on the protection scope of this application. For those skilled in the art, any other implementation methods expanded based on the scheme of this application without creative work belong to the protection scope of this application.

[0076] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0077] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0078] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0080] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for training a speech recognition model, characterized in that: The method comprises: Inputting audio feature data of the first audio file into the speech recognition model; wherein the audio feature data is obtained by masking processing, and the masking processing is used to mask the audio feature data with a receptive field as an interval; the masking length is less than or equal to 1 / n of the receptive field, and n is a positive integer; Based on the audio feature data, a speech recognition model is trained.

2. The method according to claim 1, characterized in that The masking process is performed on the audio feature data, comprising: determining at least two of the following masking parameters: a masking start point, a masking length, and a masking end point; The audio feature data within the data range indicated by the masking parameter is replaced.

3. The method according to claim 2, characterized in that The replacing operation of the audio feature data within the data range indicated by the masking parameter comprises: The masking parameter indicates that the audio feature data within the data range is replaced with a specified value or a random number.

4. The method according to claim 2, characterized in that: Filtering data that needs masking processing from the audio feature data includes: The audio feature data are sampled at equal intervals, and the size of each sampling area is the same as the receptive field size.

5. The method according to claim 4, characterized in that The sampling interval is the receptive field size.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Filtering out the first audio feature file from the training sample set, and using the remaining audio files as the second audio files; The training samples used to train the speech recognition model also include audio feature data of the second audio file.

7. A speech recognition method, characterized in that: The method comprises: Get the speech to be recognized; The speech to be recognized is input into the speech recognition model as described in any one of claims 1 to 6 to obtain a speech recognition result.

8. An electronic device, characterized in that: include: A display, a user interface for displaying audio file data; A memory for storing executable instructions of the processor; A processor, configured to execute the executable instructions to implement the method according to any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images