Speech recognition method and apparatus

Abstract

The application provides a speech recognition method and device, and the scheme of the application comprises the following steps: determining the acoustic characteristics of a speech signal, determining the first string sequence corresponding to the speech signal based on the acoustic characteristics of the speech signal, determining the non-Chinese string sequence in the first string sequence, performing mask processing on the characters in the non-Chinese string sequence in the first string sequence to obtain a second string sequence, and determining the speech recognition result of the speech signal based on the acoustic characteristics of the speech signal and the second string sequence.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a speech recognition method and apparatus. Background Technology

[0002] Speech recognition is the technology of converting speech signals into text or commands. The application areas of speech recognition technology are increasing.

[0003] However, if the speech signal to be recognized is an audio signal that mixes Chinese and foreign languages, recognition errors are likely to occur, resulting in low accuracy of speech recognition. Summary of the Invention

[0004] On one hand, embodiments of this application provide a speech recognition method, including:

[0005] Determine the acoustic characteristics of the speech signal;

[0006] Based on the acoustic features of the speech signal, a first string sequence corresponding to the speech signal is determined;

[0007] Identify the non-Chinese string sequences in the first string sequence, and mask the characters in the non-Chinese string sequences in the first string sequence to obtain the second string sequence;

[0008] Based on the acoustic features of the speech signal and the second string sequence, the speech recognition result of the speech signal is determined.

[0009] One possible implementation also includes:

[0010] If the first string sequence does not contain a non-Chinese string sequence, the speech recognition result of the speech signal is determined based on the first string sequence.

[0011] In another possible implementation, the masking of characters in the non-Chinese string sequence of the first string sequence includes:

[0012] Mask the spaces in the non-Chinese string sequences within the first string sequence.

[0013] In another possible implementation, determining the speech recognition result of the speech signal based on the acoustic features of the speech signal and the second string sequence includes:

[0014] Based on the acoustic features of the speech signal and the second string sequence, the third string sequence corresponding to the speech signal is determined using the first recognition module;

[0015] The first recognition module is trained using mask sequence samples corresponding to multiple string sequence samples labeled with actual string sequences. The mask sequence samples are obtained by masking at least one character in the string sequence samples.

[0016] In yet another possible implementation, determining the acoustic features of the speech signal includes:

[0017] The acoustic features of the speech signal are determined using a feature recognition module;

[0018] Determining the first string sequence corresponding to the speech signal based on its acoustic features includes:

[0019] Based on the acoustic features of the speech signal, a first string sequence corresponding to the speech signal is determined using a second recognition module. The second recognition module is trained using multiple speech signal samples labeled with actual string sequences.

[0020] In another possible implementation, the string sequence sample is a string sequence obtained by processing the speech signal sample through the feature recognition module and the second recognition module.

[0021] In another possible implementation, at least one character in the string sequence sample is masked, including:

[0022] If the string sequence sample does not include non-Chinese string sequences, at least one space in the string sequence sample is randomly masked.

[0023] If the string sequence sample includes non-Chinese string sequences, the spaces in the non-Chinese string sequences in the string sequence sample are masked, and the spaces in the Chinese string sequences of the string sequence sample and the foreign characters in the non-Chinese string sequences of the string sequence sample are randomly masked.

[0024] In another possible implementation, the feature recognition module, the first recognition module, and the second recognition module are trained in the following manner:

[0025] Obtain multiple speech signal samples labeled with actual string sequences;

[0026] The speech signal sample is input into the feature recognition module to be trained to obtain the acoustic features of the speech signal sample;

[0027] The acoustic features of the speech signal sample are input into the second recognition module to be trained to obtain the string sequence sample output by the second recognition module;

[0028] At least one character in the string sequence sample is masked to obtain a masked sequence sample.

[0029] The mask sequence sample is input into the first recognition module to be trained, and the string recognition result output by the first recognition module is obtained;

[0030] If, based on the string recognition result corresponding to the speech signal sample and the actual labeled string sequence, it is determined that the training termination condition is not met, the parameters of the feature recognition module, the first recognition module, and the second recognition module are adjusted, and the operation of inputting the speech signal sample into the feature recognition module to be trained is returned.

[0031] If the string recognition result corresponding to the speech signal sample and the actual labeled string sequence determine that the training termination condition is met, the training ends.

[0032] In another aspect, this application also provides a voice recognition device, comprising:

[0033] Feature determination unit, used to determine the acoustic features of speech signals;

[0034] The first recognition unit is used to determine the first string sequence corresponding to the speech signal based on the acoustic features of the speech signal;

[0035] The masking unit is used to determine the non-Chinese string sequence in the first string sequence, and to mask the characters in the non-Chinese string sequence in the first string sequence to obtain the second string sequence.

[0036] The second recognition unit is used to determine the speech recognition result of the speech signal based on the acoustic features of the speech signal and the second string sequence.

[0037] In one possible implementation, the device further includes:

[0038] The result determination unit is used to determine the speech recognition result of the speech signal based on the first string sequence if the first string sequence does not contain a non-Chinese string sequence. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0040] Figure 1This paper illustrates a flowchart of a speech recognition method provided in an embodiment of this application.

[0041] Figure 2 This illustration shows another flowchart of the speech recognition method provided in an embodiment of this application;

[0042] Figure 3 This paper illustrates a flowchart of a model training method provided in an embodiment of this application.

[0043] Figure 4 This paper illustrates a schematic diagram of the principle framework for model training in an embodiment of this application.

[0044] Figure 5 This illustration shows another flowchart of the speech recognition method provided in an embodiment of this application;

[0045] Figure 6 This illustration shows a schematic diagram of the implementation principle framework of the speech recognition method provided in an embodiment of this application;

[0046] Figure 7 This paper shows a schematic diagram of the composition structure of a speech recognition device provided in an embodiment of this application;

[0047] Figure 8 A schematic diagram of the component architecture of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0048] The solution proposed in this application is applicable to any scenario requiring speech signal recognition, thereby improving the accuracy of recognizing speech signals containing non-Chinese characters.

[0049] For example, the solution in this application embodiment can be applied to electronic devices involving voice recognition. This electronic device can be a server providing voice recognition services, or it can be a user terminal device with voice recognition capabilities, such as a mobile phone, laptop, smart speaker, or smart home appliance, etc., without limitation. Of course, the solution in this application can also be applied to applications in electronic devices that require voice recognition services, without limitation.

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] like Figure 1This document illustrates a flowchart of a speech recognition method provided in an embodiment of this application. The method of this embodiment can be applied to the aforementioned electronic devices, and may include:

[0052] S101, Determine the acoustic characteristics of the speech signal.

[0053] Among them, acoustic features are the characteristics of speech signals in terms of sound.

[0054] For example, acoustic features include, but are not limited to, features related to speech information recognition such as the frequency, amplitude, and pitch of the speech signal, and there are no restrictions on these features.

[0055] The acoustic features of a speech signal can be determined by any method of extracting acoustic features from the speech signal, without any restrictions.

[0056] S102, Based on the acoustic features of the speech signal, determine the first string sequence corresponding to the speech signal.

[0057] The first string sequence is a string sequence obtained by combining the acoustic features of the speech signal to perform speech recognition.

[0058] Depending on the type of speech signal, the first string sequence may contain one or more characters. Accordingly, since the speech signal can be input Chinese speech, foreign language speech, or a speech signal mixed with Chinese and foreign languages, the characters contained in the first string sequence may include one or more of the following: Chinese characters, foreign language characters, spaces, and other special characters, without limitation.

[0059] The determination of the first string sequence can be based on the acoustic features of the speech signal, and any method of speech recognition can be used without restriction.

[0060] S103, determine the non-Chinese string sequence in the first string sequence, and mask the characters in the non-Chinese string sequence in the first string sequence to obtain the second string sequence.

[0061] The non-Chinese text string sequence in the first string sequence refers to the string sequence composed of non-Chinese characters in the first string sequence.

[0062] The non-Chinese string sequence contains only non-Chinese characters and spaces between them. A non-Chinese string sequence may be part or all of the first string sequence. If the first string sequence does not contain Chinese text, then the first string sequence may consist entirely of non-Chinese string sequences.

[0063] For example, taking Chinese-English mixed speech signals as an example, the first character string sequence recognized from the speech signal will not only contain Chinese characters, but may also contain English words. Then, in the first character string sequence, a sequence composed of at least one consecutive English word and the space characters between the at least one English word is a non-Chinese character string sequence.

[0064] For example, the first character string sequence recognized from the speech signal is: "今~天~~天气~怎~么~样Go_~night~~and~~~lo_~~~_jing", where "~" represents a space, and the non-Chinese character string sequence in this first character string sequence is "Go_~night~~and~~~lo_~~~_jing".

[0065] Of course, in practical applications, if the speech signal input by the user is a mixture of multiple segments of Chinese and multiple segments of English, then the first character string sequence may also contain multiple segments of non-Chinese character string sequences.

[0066] In the present application, masking characters in each non-Chinese character string sequence in the first character string sequence refers to masking some characters in the non-Chinese character string sequence, so that the masked characters can be re-recognized subsequently, so as to reduce the situation that foreign characters are misrecognized.

[0067] Wherein, there may be multiple ways to perform masking processing on characters in a non-Chinese character string sequence. For example, random masking processing may be performed on characters in the non-Chinese character string sequence, or masking processing may be performed on specific characters in the non-Chinese character string sequence.

[0068] Through research by the inventors of the present application, it is found that when currently recognizing speech signals containing foreign languages, foreign characters in the speech signal are often misrecognized as spaces, which leads to foreign character deletion errors in the recognized non-Chinese character string sequence. For example, for a speech signal containing English, some English characters are likely to be misrecognized as characters.

[0069] Based on this, in a possible implementation, in order to subsequently focus on recognizing spaces in the non-Chinese character string sequence, so as to recognize foreign characters that are misrecognized as spaces and correct the situation where text characters are misrecognized as spaces, the present application can perform masking processing on spaces in the non-Chinese character string sequence in the first character string sequence.

[0070] Wherein, masking processing for characters such as spaces or foreign letters in a non-Chinese character string may be replacing the characters with a set symbol, which is generally a specific symbol distinguished from Chinese characters, foreign characters and spaces. For example, the set symbol may be "#", and of course, it may also be set to other symbols as required, which is not limited herein.

[0071] To facilitate distinction, in the present application, the string sequence obtained after mask processing on the non-Chinese character string sequence in the first string sequence is referred to as a second string sequence.

[0072] By way of example, the first string sequence is: "Jin ~ Tian ~ ~ Tian Qi ~ Zen ~ Me ~ Yang Go_ ~ night ~ ~ and ~ ~ ~ lo_ ~ ~ ~ _jing".

[0073] An example is described in which spaces in the non-Chinese character string sequence in the first string sequence are replaced with the mask symbol "#". The spaces in "Go_ ~ night ~ ~ and ~ ~ ~ lo_ ~ ~ ~ _jing" in the first string sequence are replaced with "#", and therefore the second string sequence obtained after mask processing on the first string sequence is "Jin ~ Tian ~ ~ Tian Qi ~ Zen ~ Me ~ Yang Go_#night##andlo__jing".

[0074] S104: determining a speech recognition result of the speech signal based on the acoustic features of the speech signal and the second string sequence.

[0075] After the present application performs speech recognition on the speech signal based on the acoustic features of the speech signal, the recognized first string sequence is not directly determined as the speech recognition result of the speech signal. Instead, speech recognition needs to be performed again based on the second string sequence, and the recognized result is taken as the speech recognition result of the speech signal.

[0076] Any speech recognition method may be used for performing speech recognition by combining the acoustic features of the speech signal and the second string sequence, which is not limited herein.

[0077] In an alternative mode, considering that if the speech signal only contains Chinese speech information, there will be no problems such as misrecognizing foreign characters as spaces, and the accuracy of the recognition result obtained by performing speech recognition on the speech signal is relatively high. Based on this, if the first string sequence does not include a non-Chinese character string sequence, the speech recognition result of the speech signal can be determined based on the first string sequence.

[0078] For example, the first string sequence is determined as the speech recognition result. In another example, considering that the first string sequence may involve merging between some characters or some other formatting processing, the string sequence obtained after performing formatting processing on the first string sequence may be determined as the speech recognition result.

[0079] As can be seen from the above, in this embodiment of the application, after determining the first string sequence corresponding to the speech signal based on the acoustic features of the speech signal, the characters in the non-Chinese string sequence in the first string sequence are masked. On this basis, in the process of speech recognition, combined with the acoustic features of the speech signal and the second string sequence obtained by masking, the effective recognition of non-Chinese content in the speech signal can be strengthened, and the situation of erroneous deletion of non-Chinese characters due to the mixing of non-Chinese characters in the speech signal can be reduced, thereby improving the accuracy of speech recognition of speech signals containing non-Chinese characters.

[0080] It is understood that there can be various specific implementations of speech recognition based on the acoustic features of speech signals and the second string sequence, and this application does not impose any restrictions on this.

[0081] In order to more accurately identify the speech recognition result of the speech signal based on the acoustic features of the speech signal and the second string sequence, this application can also pre-train a recognition module for speech recognition. For ease of distinction, the recognition module for speech recognition based on the acoustic model of the speech signal and the second string sequence is referred to as the first recognition module.

[0082] The first recognition module is trained using mask sequence samples corresponding to multiple string sequence samples labeled with actual string sequences.

[0083] The string sequence sample can be a manually selected string sequence. For example, it can be obtained from a database or different data platforms, containing string sequences that were correctly identified from different speech signals in the past. These string sequences may include correctly identified string sequences as well as incorrectly identified string sequences. Correspondingly, the actual string sequence contained in the speech signal is the actual string sequence that needs to be labeled for the string sequence identified from that speech signal.

[0084] The string sequence sample can also be a string sequence obtained from multiple speech signal samples through speech recognition. For example, by extracting the acoustic features of the speech signal samples, performing speech recognition on the speech signal samples based on the acoustic features, and determining the string sequence as the string sequence sample for training. The correct string sequence actually corresponding to the speech signal sample is the actual string sequence labeled by this string sequence sample.

[0085] The masked sequence sample is obtained by masking at least one character in the string sequence sample.

[0086] There are several ways to mask at least one character in a string sequence sample. For example, characters in the string sequence sample can be randomly masked, that is, one or more characters can be randomly selected from the string sequence sample for masking. Another example is to mask the whitespace portions contained in the string sequence sample, or to randomly select whitespace for masking.

[0087] In order for the model to accurately identify the real characters corresponding to the masked characters, this application needs to perform some random masking on the other parts of the characters, in addition to masking the spaces in the non-Chinese character sequence.

[0088] For example, in one possible implementation, if the string sequence sample does not contain non-Chinese string sequences, at least one space in the string sequence sample is randomly masked. That is, if the string sequence sample only contains Chinese characters and spaces, then only a portion of the spaces in the string sequence sample need to be randomly selected for masking.

[0089] If the string sequence sample includes non-Chinese string sequences, the spaces in the non-Chinese string sequences in the string sequence sample are masked, and the spaces in the Chinese string sequences and the foreign characters in the non-Chinese string sequences in the string sequence sample are randomly masked.

[0090] In particular, spaces in non-Chinese string sequences must be masked. This is because spaces in non-Chinese string sequences may be foreign characters that have been misidentified. Therefore, in order for the first recognition module to focus on identifying spaces that may be misidentified, spaces need to be masked so that the first recognition module can focus on re-identifying the masked characters during the training process.

[0091] However, for spaces within Chinese string sequences and other characters besides spaces in non-Chinese string sequences, some characters can be randomly selected for masking to improve the recognition accuracy of the first recognition module through training.

[0092] It is understandable that any supervised training method can be used to train the first recognition module, without any restrictions.

[0093] Accordingly, based on the acoustic features of the speech signal and the second string sequence, the third string sequence corresponding to the speech signal can be determined using the first recognition module.

[0094] Based on this, the third string sequence can be used as the speech recognition result of the speech signal, or the third string sequence can be formatted and the formatted third string sequence can be used as the speech recognition result.

[0095] The first recognition module can be an independent model.

[0096] In particular, since the determination of the first string sequence is also based on the recognition of the speech signal by the model, the first recognition module and the second recognition module that recognizes the first string sequence can be used as a whole as a recognition model, or as part of a recognition model.

[0097] The following example illustrates how to identify the first string sequence from a speech signal using a model and how to perform speech recognition on the second string sequence after masking using the model.

[0098] like Figure 2 As shown, it illustrates another flowchart of the speech recognition method provided in this application embodiment. The method of this embodiment may include:

[0099] S201, the acoustic features of the speech signal are determined using the feature recognition module.

[0100] The feature recognition module can be a model used to extract acoustic features from speech signals. This feature recognition module can also be combined with the subsequent first and second recognition modules to form a speech recognition model; therefore, the feature recognition module is a part of the speech recognition model.

[0101] In this application, the feature recognition module can be obtained by supervised training using multiple speech signals labeled with actual acoustic features; or it can be obtained by training multiple speech signal samples labeled with actual string sequences together with the subsequent first recognition module and second recognition module.

[0102] S202, based on the acoustic features of the speech signal, the second recognition module is used to determine the first string sequence corresponding to the speech signal.

[0103] The second recognition module is trained using multiple speech signal samples labeled with actual string sequences.

[0104] For example, a second recognition module can be trained using supervised training methods, utilizing the acoustic features of multiple speech signal samples labeled with actual string sequences. Alternatively, a supervised training method can be used to train the feature recognition module, the second recognition module, and the subsequent first recognition module together using the acoustic features of multiple speech signal samples labeled with actual string sequences, ultimately obtaining these trained recognition modules.

[0105] It should be noted that, for ease of distinction, this application refers to the recognition module for speech recognition based on the second string sequence mentioned above as the first recognition module, and the model for speech recognition of speech signals here as the second recognition module.

[0106] S203, determine the non-Chinese string sequence in the first string sequence, and mask the characters in the non-Chinese string sequence in the first string sequence to obtain the second string sequence.

[0107] This step can be referred to in the relevant description of the previous embodiments, and will not be repeated here.

[0108] For example, spaces in the non-Chinese string sequences of the first string sequence can be masked to obtain the second string sequence.

[0109] S204, based on the acoustic features of the speech signal and the second string sequence, the third string sequence corresponding to the speech signal is determined using the first recognition module.

[0110] The first recognition module is trained using mask sequence samples corresponding to multiple string sequence samples labeled with actual string sequences. These mask sequence samples are obtained by masking at least one character in the string sequence sample.

[0111] The first recognition module can be trained using the training method mentioned above.

[0112] In one possible implementation, the string sequence sample corresponding to the mask sequence sample used to train the first recognition module can be a string sequence obtained by processing a speech signal sample through a feature recognition module and a second recognition module. Here, the speech signal sample is the speech signal sample used to train the feature recognition module or the second recognition module. Correspondingly, the actual string sequence labeled by this speech signal sample can be used as the actual string sequence labeled by the string sequence sample used to train the first recognition module.

[0113] In this embodiment, after the acoustic features of the speech signal are extracted by the feature recognition module, the second recognition module is used to perform speech recognition based on the acoustic features to obtain the first string sequence of the speech signal. The trained model is used to perform preliminary recognition of the speech signal, which can improve the accuracy of the recognized speech signal and is beneficial to the accuracy of the recognition of the first string sequence.

[0114] Based on this, this application performs masking on the characters in the non-Chinese string sequence of the first string sequence to obtain the second string sequence. Then, the first recognition module combines the acoustic features with the second string sequence for speech recognition. Since the first recognition module is also trained using the masked sequence samples corresponding to the actual string sequence samples, the first recognition module can more effectively correct the masked and misidentified characters in the second string sequence, thereby reducing the situation where foreign characters are incorrectly identified as spaces. As a result, the third string sequence obtained from the final speech signal recognition is more accurate.

[0115] To facilitate understanding of the training process of the feature recognition module, the first recognition module, and the second recognition module in this application, the following explanation will use the synchronous training of these three recognition modules as an example, combined with a training method.

[0116] like Figure 3 The diagram illustrates an implementation flow of the training feature recognition module, the first recognition module, and the second recognition module in an embodiment of this application. The method of this embodiment may include:

[0117] S301, obtain multiple speech signal samples labeled with actual string sequences.

[0118] The speech signal samples are the speech signals used as training samples. The actual string sequences labeled on the speech signal samples are the actual text content contained in the speech signal samples.

[0119] S302, for each speech signal sample, input the speech signal sample into the feature recognition module to be trained to obtain the acoustic features of the speech signal sample.

[0120] For example, the feature recognition module can be a feature encoder or a network model, without any restrictions.

[0121] S303, input the acoustic features of the speech signal sample into the second recognition module to be trained, and obtain the string sequence sample output by the second recognition module.

[0122] In this application, the string sequence determined by the second recognition module based on the acoustic features of the speech signal sample during the training process is called the string sequence sample. The first recognition module can be trained based on the string sequence sample.

[0123] S304, perform masking on at least one character in the string sequence sample to obtain a masked sequence sample.

[0124] It is understandable that each speech signal sample can be processed into a string sequence sample after steps S302 and S303.

[0125] Wherein, for the masking method of at least one character in the string sequence sample, reference may be made to the method mentioned above.

[0126] For ease of understanding, one situation is taken as an example and described with specific examples.

[0127] For example, if the string sequence sample does not include a non-Chinese character string sequence, random masking processing can be performed on at least one space in the string sequence sample. For example, take the string sequence sample "I~to~day~~feel~~good" as an example, where "~" represents a space. Assuming that the masking processing is to replace the character with "#", some spaces can be randomly selected for masking processing in the string sequence sample, and the masked sequence sample obtained after masking processing can be "I#to~day##feel~#good", or it can also be "I~to#day~~feel##good", of course, other possibilities also exist.

[0128] For the case where the string sequence sample includes a non-Chinese character string sequence, reference may be made to Figure 4 , which shows a schematic diagram of a principle framework for model training in the present application.

[0129] It can be seen from Figure 4 that after the speech signal sample is processed by the feature recognition module to extract acoustic features, the acoustic features are input into the second recognition module, and the string sequence sample recognized by the second recognition module is obtained as "to~day~~weather~ho~w~is~it Go_~night~~and~~~lo_~~~_ing". This string sequence sample contains a non-Chinese character string sequence composed of English words.

[0130] Assuming that for a string sequence sample containing a non-Chinese character string sequence, take the example of masking the spaces in the non-Chinese character string sequence of the string sequence sample, and randomly masking the spaces in the Chinese character string sequence of the string sequence sample and the foreign characters in the non-Chinese character string sequence of the string sequence sample. Then for the Figure 4 string sequence sample output by the second recognition module, the masked sequence sample obtained after masking processing can be: "to#day~~weather#ho~w~is~it##_#night##andlo__ing".

[0131] It can be seen from Figure 4It can be known from the masked sequence sample obtained after masking processing that all spaces in the non-Chinese character sequence "Go_~night~~and~~~lo_~~~_jing" in the string sequence sample are replaced with the mask symbol "#", and in addition, the English word "Go" is randomly selected from the non-Chinese character sequence for masking processing. Part of the spaces are randomly selected for masking processing in the Chinese character sequence part of the string sequence sample, for example, the space in "Jin~tian" is replaced with the mask symbol "#".

[0132] It can be understood that when performing masking processing on characters other than spaces in a non-Chinese character sequence, if a plurality of characters form a foreign word (such as an English word), then the foreign word (e.g. Figure 4 "Go" in this case) can be replaced with only one mask symbol, so as to reduce the complexity of masking processing and improve training efficiency.

[0133] S305, input the masked sequence sample into the first recognition module to be trained, and obtain the string recognition result output by the first recognition module.

[0134] Wherein, the string recognition result is the string sequence recognized by the first recognition module.

[0135] S306, if it is determined that the training end condition is not satisfied based on the string recognition result corresponding to each voice signal sample and the labeled actual string sequence, adjust the parameters of the feature recognition module, the first recognition module and the second recognition module, and return to step S302.

[0136] For example, according to a set loss function, the loss function value is calculated by using the actual string sequence labeled for each voice signal sample and the string sequence output by the first recognition module, and if the loss function value converges, it is determined that the training end condition is satisfied.

[0137] For another example, the prediction accuracy is determined based on the actual string sequence labeled for each voice signal sample and the string sequence output by the first recognition module, and if the prediction accuracy meets the requirement, it is determined that the training end condition is satisfied.

[0138] Of course, there may be other possibilities for the training end condition, which is not limited herein.

[0139] If the training end condition is not satisfied yet, it is necessary to adjust the parameters of these three modules and return to continue training until the training end condition is satisfied.

[0140] In the present application, the first recognition module and the second recognition module may be decoders, or may be network models, which is not limited herein.

[0141] S307. If the string recognition result corresponding to the speech signal sample and the actual string sequence labeled are determined to meet the training termination condition, the training ends.

[0142] It is understandable that the trained feature recognition module, the second recognition module, and the first recognition module are each a separate model, and the three modules as a whole can be regarded as a speech recognition model.

[0143] As can be seen, in order for the first recognition module to identify characters such as spaces or letters that are misidentified in the string sequence sample, this application performs masking processing on the string sequence sample, and continuously identifies and trains the masked sequence sample obtained by the first recognition module in combination with the acoustic feature masking processing, so that the trained first recognition module can accurately identify foreign characters that are misidentified as spaces.

[0144] To facilitate understanding of the solution in this application, the speech recognition method of this application will be described below with reference to one implementation method, such as... Figure 5 As shown, it illustrates another flowchart of the speech recognition method provided in this application embodiment. The method of this embodiment may include:

[0145] S501 uses a feature recognition module to determine the acoustic features of the speech signal to be recognized.

[0146] S502, based on the acoustic features of the speech signal, the second recognition module is used to determine the first string sequence corresponding to the speech signal.

[0147] S503, if the first string sequence does not contain a non-Chinese string sequence, determine the speech recognition result of the speech signal based on the first string sequence.

[0148] It is understandable that if the first string sequence does not contain non-Chinese string sequences, then since the recognition technology for speech signals containing only Chinese characters is already relatively mature, the Chinese character sequence recognized by the speech signal can be considered accurate, and no further processing is needed. The speech recognition result of the speech signal can be determined based on the first string sequence.

[0149] The first string sequence can be directly identified as the speech recognition result.

[0150] In one possible implementation, considering that there may be some extra spaces or repeated characters in the first string sequence of Chinese characters recognized from the speech signal, after recognizing the first string sequence, some formatting processing such as deduplication or space deletion can be performed on the first string sequence, and the formatted string sequence is determined as the text result of speech signal recognition.

[0151] For example, if the recognized first character string sequence is "Jin~Tian Tian~Qi Zenme~Yang", after removing spaces and some obvious repeated characters, the speech recognition result obtained is "How is the weather today".

[0152] S504, if the first character string sequence includes a non-Chinese character string sequence, performing masking processing on spaces in the non-Chinese character string sequence in the first character string sequence to obtain a second character string sequence.

[0153] It can be known from the foregoing introduction that after the first recognition module is trained, since the first recognition module can accurately recognize foreign language characters that are erroneously recognized as spaces, when the first recognition module is applied for speech recognition, it is only necessary to mask spaces that may be misrecognized in the non-Chinese character string sequence, so that the first recognition module can focus on recognizing the masked characters.

[0154] S505, based on the acoustic features of the speech signal and the second character string sequence, determining a third character string sequence corresponding to the speech signal by using the first recognition module.

[0155] Similar to the foregoing step S503, the present application can determine the third character string sequence as the speech recognition result of the speech signal, therefore, all speech recognition processing is completed after the third character string sequence is recognized.

[0156] Of course, considering that there may also be extra spaces, repeated words or characters that need to be merged in the third character string sequence, the present application also needs to process the third character string sequence in accordance with the set character string formatting processing rules, and finally obtain the speech recognition result of the speech signal.

[0157] In combination with Figure 6 description, Figure 6 shows a schematic block diagram of an implementation process of the speech recognition method in an embodiment of the present application.

[0158] It can be seen from Figure 6 that after the speech signal to be recognized passes through the feature recognition module and the second feature recognition module, the first character string sequence "Jin~Tian~~Tianqi~Zen~me~Yang Go_~night~~and~~~lo_~~~_jing" can be obtained.

[0159] according to the method of this embodiment, only spaces in the English part of the first character string sequence are masked, to obtain the second character string sequence: "Jin~Tian~~Tianqi~Zen~me~Yang Go_#night##andlo__jing".

[0160] Inputting the second character string sequence into the first recognition module can obtain the third character string sequence as: "Jin~Tian~~TianQi~Zen~me~YangGo__od~night~~and~i~lo__ve~bei__jing".

[0161] By comparing the third character string sequence and the first character string sequence, it can be seen that "_od" is erroneously recognized as a space in the first character string sequence, and after re-recognition by the first recognition module, it can be determined that this space is "_od". Similarly, one space between "and" and "lo" in the first character string sequence should be "i". Several other differences between the third character string sequence and the first character string sequence are similar, and will not be repeated here.

[0162] It can be understood that after obtaining the third character string sequence, since some words in the third character string sequence need to be merged, for example, in the third character string sequence, adjacent "_" between letters indicates the root of a word, which means that it needs to be merged with the preceding or following word. Meanwhile, some redundant spaces in the third character string sequence also need to be deleted. Based on this, by performing corresponding formatting processing on the third character string sequence, the final text recognition result of the speech signal can be obtained as: "Jin~Tian~~TianQi~Zen~me~YangGood night and i love beijing".

[0163] After obtaining the third character string sequence, the present application may impose no limitation on the specific process of determining the speech recognition result based on the third character string sequence.

[0164] Corresponding to the speech recognition method of the present application, the present application further provides a speech recognition apparatus.

[0165] As Figure 7 , which shows a schematic diagram of a composition structure of the speech recognition apparatus provided by the embodiment of the present application, the apparatus of this embodiment may include:

[0166] a feature determining unit 701, configured to determine acoustic features of a speech signal;

[0167] a first recognition unit 702, configured to determine a first character string sequence corresponding to the speech signal based on the acoustic features of the speech signal;

[0168] a mask processing unit 703, configured to determine a non-Chinese character string sequence in the first character string sequence, perform mask processing on characters in the non-Chinese character string sequence in the first character string sequence, and obtain a second character string sequence;

[0169] a second recognition unit 704, configured to determine a speech recognition result of the speech signal based on the acoustic features of the speech signal and the second character string sequence.

[0170] In yet another possible implementation, the device further includes:

[0171] The result determination unit is used to determine the speech recognition result of the speech signal based on the first string sequence if the first string sequence does not contain a non-Chinese string sequence.

[0172] In another possible implementation, the mask processing unit includes:

[0173] The masking subunit is used to mask spaces in the non-Chinese string sequences of the first string sequence.

[0174] In yet another possible implementation, the second identification unit includes:

[0175] The second recognition subunit is used to determine the third string sequence corresponding to the speech signal based on the acoustic features of the speech signal and the second string sequence using the first recognition module. The first recognition module is trained using mask sequence samples corresponding to multiple string sequence samples labeled with actual string sequences. The mask sequence samples are obtained by masking at least one character in the string sequence samples.

[0176] In yet another possible implementation, the feature determining unit includes:

[0177] The feature determination subunit is used to determine the acoustic features of the speech signal using the feature recognition module;

[0178] The first identification unit includes:

[0179] The first recognition subunit is used to determine the first string sequence corresponding to the speech signal based on the acoustic features of the speech signal using the second recognition module. The second recognition module is trained using multiple speech signal samples labeled with actual string sequences.

[0180] In another possible implementation, the string sequence sample corresponding to the mask sequence sample used to train the first recognition module in the second recognition subunit is a string sequence obtained by processing the speech signal sample used to train the second recognition module through the feature recognition module and the second recognition module.

[0181] In another possible implementation, the mask sequence sample in the second identification subunit is obtained by the following masking process:

[0182] If the string sequence sample does not include non-Chinese string sequences, at least one space in the string sequence sample is randomly masked.

[0183] If the string sequence sample includes non-Chinese string sequences, the spaces in the non-Chinese string sequences in the string sequence sample are masked, and the spaces in the Chinese string sequences of the string sequence sample and the foreign characters in the non-Chinese string sequences of the string sequence sample are randomly masked.

[0184] In another possible implementation, the device further includes a model training unit, which is used to train the feature recognition module, the first recognition module, and the second recognition module in the following manner:

[0185] Obtain multiple speech signal samples labeled with actual string sequences;

[0186] The speech signal sample is input into the feature recognition module to be trained to obtain the acoustic features of the speech signal sample;

[0187] The acoustic features of the speech signal sample are input into the second recognition module to be trained to obtain the string sequence sample output by the second recognition module;

[0188] At least one character in the string sequence sample is masked to obtain a masked sequence sample.

[0189] The mask sequence sample is input into the first recognition module to be trained, and the string recognition result output by the first recognition module is obtained;

[0190] If, based on the string recognition result corresponding to the speech signal sample and the actual labeled string sequence, it is determined that the training termination condition is not met, the parameters of the feature recognition module, the first recognition module, and the second recognition module are adjusted, and the operation of inputting the speech signal sample into the feature recognition module to be trained is returned.

[0191] If the string recognition result corresponding to the speech signal sample and the actual labeled string sequence determine that the training termination condition is met, the training ends.

[0192] Furthermore, this application also provides an electronic device, such as Figure 8 As shown, it illustrates a schematic diagram of the composition structure of the electronic device. The electronic device can be any type of electronic device, and the electronic device includes at least a processor 801 and a memory 802.

[0193] The processor 801 is used to execute the speech recognition method in any of the above embodiments.

[0194] The memory 802 is used to store programs required by the processor to perform operations.

[0195] It is understood that the electronic device may also include a display unit 803 and an input unit 804.

[0196] Of course, the electronic device can also have more than Figure 8 There are no restrictions on the number of more or fewer components.

[0197] On the other hand, this application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the speech recognition method as described in any of the above embodiments.

[0198] This application also proposes a computer program comprising computer instructions stored in a computer-readable storage medium. When run on an electronic device, the computer program performs the speech recognition method as described in any of the above embodiments.

[0199] It is understood that in this application, the terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar parts and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that illustrated herein.

[0200] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Furthermore, the features described in the various embodiments of this specification can be substituted or combined with each other, enabling those skilled in the art to implement or use this application. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0201] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0202] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0203] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A speech recognition method, comprising: Determine the acoustic characteristics of the speech signal; Based on the acoustic features of the speech signal, a first string sequence corresponding to the speech signal is determined; Identify the non-Chinese string sequences in the first string sequence, and mask the spaces in the non-Chinese string sequences in the first string sequence to obtain the second string sequence; Based on the acoustic features of the speech signal and the second string sequence, the speech recognition result of the speech signal is determined by the first recognition module. The first recognition module is trained using mask sequence samples corresponding to multiple string sequence samples labeled with actual string sequences. The string sequence samples are obtained by speech recognition of the speech signal samples based on the acoustic features of the speech signal samples.

2. The method according to claim 1, further comprising: If the first string sequence does not contain a non-Chinese string sequence, the speech recognition result of the speech signal is determined based on the first string sequence.

3. The method according to claim 1, wherein the speech recognition result of the speech signal is the third string sequence corresponding to the speech signal; The masked sequence sample is obtained by masking at least one character in the string sequence sample.

4. The method according to claim 3, wherein determining the acoustic features of the speech signal comprises: The acoustic features of the speech signal are determined using a feature recognition module; Determining the first string sequence corresponding to the speech signal based on its acoustic features includes: Based on the acoustic features of the speech signal, a first string sequence corresponding to the speech signal is determined using a second recognition module. The second recognition module is trained using multiple speech signal samples labeled with actual string sequences.

5. The method according to claim 4, wherein the string sequence sample is a string sequence obtained by processing the speech signal sample through the feature recognition module and the second recognition module.

6. The method according to claim 3, wherein at least one character in the string sequence sample is masked, comprising: If the string sequence sample does not include non-Chinese string sequences, at least one space in the string sequence sample is randomly masked. If the string sequence sample includes non-Chinese string sequences, the spaces in the non-Chinese string sequences in the string sequence sample are masked, and the spaces in the Chinese string sequences of the string sequence sample and the foreign characters in the non-Chinese string sequences of the string sequence sample are randomly masked.

7. The method according to claim 5 or 6, wherein the feature recognition module, the first recognition module, and the second recognition module are trained in the following manner: Obtain multiple speech signal samples labeled with actual string sequences; The speech signal sample is input into the feature recognition module to be trained to obtain the acoustic features of the speech signal sample; The acoustic features of the speech signal sample are input into the second recognition module to be trained to obtain the string sequence sample output by the second recognition module; At least one character in the string sequence sample is masked to obtain a masked sequence sample. The mask sequence sample is input into the first recognition module to be trained, and the string recognition result output by the first recognition module is obtained; If, based on the string recognition result corresponding to the speech signal sample and the actual labeled string sequence, it is determined that the training termination condition is not met, the parameters of the feature recognition module, the first recognition module, and the second recognition module are adjusted, and the operation of inputting the speech signal sample into the feature recognition module to be trained is returned. If the string recognition result corresponding to the speech signal sample and the actual labeled string sequence determine that the training termination condition is met, the training ends.

8. A voice recognition device, comprising: Feature determination unit, used to determine the acoustic features of speech signals; The first recognition unit is used to determine the first string sequence corresponding to the speech signal based on the acoustic features of the speech signal; The masking unit is used to determine the non-Chinese string sequences in the first string sequence, and to mask the spaces in the non-Chinese string sequences in the first string sequence to obtain the second string sequence. The second recognition unit is used to determine the speech recognition result of the speech signal based on the acoustic features of the speech signal and the second string sequence using the first recognition module. The first recognition module is trained using mask sequence samples corresponding to multiple string sequence samples labeled with actual string sequences. The string sequence samples are obtained by performing speech recognition on the speech signal samples based on the acoustic features of the speech signal samples.

9. The apparatus according to claim 8, further comprising: The result determination unit is used to determine the speech recognition result of the speech signal based on the first string sequence if the first string sequence does not contain a non-Chinese string sequence.

Images