Audio recognition method, device, electronic device and readable storage medium

Abstract

The present application provides an audio recognition method, device, electronic device and readable storage medium. First, a first type of noise in the audio to be recognized is filtered using a pre-constructed first filtering model to obtain a first recognized audio, and the first type of noise is non-speech noise. Then, a second type of noise in the first recognized audio is filtered using a pre-constructed second filtering model to obtain a second recognized audio, and the first recognized audio contains target speech and non-target speech, and the second type of noise belongs to non-target speech. Finally, the second recognized audio is imported into the pre-constructed recognition model, and the target speech in the second recognized audio is processed to obtain the corresponding text. This solution pre-filters non-speech noise and noise belonging to non-target speech, reduces the processing pressure of speech recognition and eliminates the interference of various types of noise, and has a better effect on the recognition of audio with a longer duration and mixed with noise.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio recognition method, device, electronic device and readable storage medium. Background Art

[0002] With the development of multimedia technology, there is a great demand for audio recognition and text conversion. The current audio recognition processing method can achieve better results when processing shorter audio, but when processing longer audio mixed with noise, the subsequent recognition is hindered due to the imperfect noise processing method, and the recognition pressure is too high due to the need to process a large amount of speech recognition conversion in the speech recognition stage. Therefore, the processing effect of longer audio mixed with noise is not good. Summary of the invention

[0003] The purpose of the present application includes, for example, providing an audio recognition method, device, electronic device and readable storage medium, which can reduce the processing pressure of speech recognition and eliminate the interference of various types of noise, and have better effect on audio recognition processing of longer duration and mixed with noise.

[0004] The embodiments of the present application can be implemented as follows:

[0005] In a first aspect, the present application provides an audio recognition method, the method comprising:

[0006] Using a pre-built first filtering model to filter a first type of noise in the audio to be recognized to obtain a first recognized audio, wherein the first type of noise is non-speech noise;

[0007] Using a pre-constructed second filtering model to filter a second type of noise in the first recognition audio to obtain a second recognition audio, the first recognition audio includes a target speech and a non-target speech, and the second type of noise belongs to the non-target speech;

[0008] The second recognition audio is imported into a pre-built recognition model, and the target speech in the second recognition audio is processed to obtain the corresponding text.

[0009] In an optional implementation, the step of filtering the first type of noise in the audio to be recognized by using the pre-built first filtering model to obtain the first recognized audio includes:

[0010] Using a pre-built first filtering model to perform volume detection on each audio frame included in the audio to be recognized;

[0011] The detected audio frames whose volume is lower than a preset volume are determined as first-type noise, and the audio frames determined as first-type noise are intercepted and filtered out from the audio to be recognized to obtain first recognized audio.

[0012] In an optional implementation, the step of filtering the second type of noise in the first recognized audio using a pre-constructed second filtering model to obtain the second recognized audio includes:

[0013] Using the pre-built second filtering model, decoding is started from the first audio frame in the first recognized audio to obtain corresponding audio features;

[0014] When the audio feature of the decoded audio frame is characterized as a non-target speech, and there is an audio frame of a set frame whose audio feature is characterized as a non-target speech starting from the audio frame, determining the audio frame of the set frame as the second type of noise, wherein the set frame exceeds a preset threshold;

[0015] The audio frame of the set frame is intercepted and filtered out from the first recognized audio, and the processing of subsequent audio frames is continued until the processing of the first recognized audio is completed.

[0016] In an optional implementation, the second filtering model stores semantic context information of the decoded audio frame;

[0017] Before the step of continuing to process subsequent audio frames until the processing of the first recognized audio is completed, the step of filtering the second type of noise in the first recognized audio using the pre-constructed second filtering model to obtain the second recognized audio further includes:

[0018] Initialize the semantic context information stored in the second filtering model.

[0019] In an optional implementation, the step of importing the second recognized audio into a pre-built recognition model and processing the target speech in the second recognized audio to obtain the corresponding text includes:

[0020] When there are audio frames in the second recognized audio whose audio features are characterized as non-target speech and the number of consecutive frames is lower than a preset threshold, determining the audio frames in the second recognized audio whose audio features are characterized as target speech;

[0021] The determined audio feature representation is obtained as text corresponding to the audio frame of the target speech.

[0022] In an optional implementation, the step of filtering the second type of noise in the first recognized audio using a pre-constructed second filtering model to obtain the second recognized audio includes:

[0023] Using a pre-built second filtering model, identify audio frames characterized as target speech and audio frames of non-target speech in the first recognition audio;

[0024] Determine a second type of noise from the recognized audio frame of the non-target speech, and determine a blank frame from the recognized audio frame of the target speech;

[0025] The second type of noise and the blank frame are intercepted and filtered out from the first recognition audio.

[0026] In an optional embodiment, the second filtering model is pre-trained based on a plurality of training samples including target speech and non-target speech;

[0027] The second type of noise in each of the training samples is marked with a start symbol and an end symbol, the second type of noise is composed of multiple audio frames of non-target speech, and the second filtering model is trained by decoding the start symbol and the end symbol of the second type of noise in each of the training samples.

[0028] In an optional implementation, the second type of noise in each of the training samples is further marked with a noise label, and the noise label identifies the type of the second type of noise.

[0029] In a second aspect, the present application provides an audio recognition device, the device comprising:

[0030] A first filtering module, configured to filter a first type of noise in the audio to be recognized by using a pre-built first filtering model to obtain a first recognized audio, wherein the first type of noise is non-speech noise;

[0031] A second filtering module, configured to filter the second type of noise in the first recognized audio using a pre-constructed second filtering model to obtain a second recognized audio, wherein the first recognized audio includes target speech and non-target speech, and the second type of noise belongs to the non-target speech;

[0032] A processing module is used to import the second recognition audio into a pre-built recognition model and process the target speech in the second recognition audio to obtain the corresponding text.

[0033] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of the aforementioned embodiments are implemented.

[0035] The beneficial effects of the embodiments of the present application include, for example:

[0036] The present application provides an audio recognition method, device, electronic device and readable storage medium. First, a first type of noise in the audio to be recognized is filtered using a pre-constructed first filtering model to obtain a first recognized audio, and the first type of noise is non-speech noise. Then, a second type of noise in the first recognized audio is filtered using a pre-constructed second filtering model to obtain a second recognized audio, and the first recognized audio contains target speech and non-target speech, and the second type of noise belongs to non-target speech. Finally, the second recognized audio is imported into the pre-constructed recognition model, and the target speech in the second recognized audio is processed to obtain the corresponding text. This solution pre-filters non-speech noise and noise belonging to non-target speech, reduces the processing pressure of speech recognition and eliminates the interference of various types of noise, and has a better effect on the recognition of audio with a longer duration and mixed with noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A schematic diagram of an application scenario of the audio recognition method provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of an audio recognition method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of processing the audio to be recognized provided in an embodiment of the present application;

[0041] Figure 4 Another schematic diagram of processing the audio to be recognized provided in an embodiment of the present application;

[0042] Figure 5 for Figure 2 Flow chart of sub-steps included in step S101;

[0043] Figure 6 for Figure 2 Flow chart of sub-steps included in step S102;

[0044] Figure 7 for Figure 2 Another flowchart of the sub-steps included in step S102;

[0045] Figure 8 for Figure 2 Flow chart of sub-steps included in step S103;

[0046] Fig. 9 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0047] Fig.10 This is a functional module block diagram of the audio recognition device provided in an embodiment of the present application.

[0048] Icons: 100 - live broadcast provider; 200 - live broadcast server; 300 - live broadcast receiver; 410 - memory; 420 - processor; 430 - audio recognition device; 431 - first filtering module; 432 - second filtering module; 433 - processing module. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0051] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0052] In the description of the present application, it should be noted that the terms "first", "second", etc., if used, are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0053] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0054] The audio recognition method provided by the present application can be applied to a variety of application scenarios, such as converting user audio into corresponding text in communication software, converting scene audio into corresponding text in monitoring scenarios, or converting the audio of the anchor side into corresponding text in live broadcast scenarios, etc. The audio recognition method provided by the present application can filter the audio information collected in a variety of application scenarios to filter out various noises therein, so as to retain the required voice part, and then perform recognition processing.

[0055] See also Figure 1 , is a schematic diagram of a possible application scenario provided in an embodiment of the present application, wherein the application scenario includes a live broadcast providing end 100, a live broadcast receiving end 300, and a live broadcast server 200 which is communicatively connected to the live broadcast providing end 100 and the live broadcast receiving end 300 respectively.

[0056] Among them, the live broadcast provider 100 can be a terminal device (such as a smart phone, tablet computer, personal computer, etc.) used by the anchor during the live broadcast, and the live broadcast receiver 300 can be a terminal device (such as a smart phone, tablet computer, personal computer, etc.) used by the audience when watching the live broadcast.

[0057] In this embodiment, the scene may also include an audio acquisition device for collecting audio, wherein the audio acquisition device may be directly installed or integrated in the live broadcast provider 100. For example, the audio acquisition device may be a microphone configured on the live broadcast provider 100, and other modules or components in the live broadcast provider 100 may receive audio information sent from the audio acquisition device via an internal bus. Alternatively, the audio acquisition device may also be independent of the live broadcast provider 100, and the two may communicate via wired or wireless means.

[0058] The live broadcast provider 100 can send the live video stream to the live broadcast server 200, and the audience can access the live broadcast server 200 through the live broadcast receiving terminal 300 to watch the live video. At the same time, the live video stream sent by the live broadcast provider carries audio information, such as audio information collected by an audio acquisition device, including the host's voice, background sound in the scene, audio emitted by the device, etc. The live broadcast server 200 can process the audio information therein to generate corresponding text, so as to facilitate the subsequent generation of corresponding subtitles, etc. Alternatively, the recognition processing of the audio information can also be performed at the live broadcast provider 100, and the live broadcast provider 100 sends the recognition result to the live broadcast server 200, which is forwarded to the live broadcast receiving terminal 300 via the live broadcast server 200.

[0059] It should be noted that the above is only one possible application scenario of the audio recognition method of the present application, and the application scenario of the audio recognition method is not limited thereto.

[0060] See also Figure 2The present application also provides an audio recognition method applicable to an electronic device, which may be the live broadcast provider 100 or the live broadcast server 200. The method steps defined in the process related to the audio recognition method may be implemented by the electronic device. Figure 2 The specific process shown is described in detail.

[0061] S101, using a pre-built first filtering model to filter a first type of noise in the audio to be recognized to obtain a first recognized audio, where the first type of noise is non-speech noise.

[0062] S102, using a pre-constructed second filtering model to filter the second type of noise in the first recognized audio to obtain a second recognized audio, wherein the first recognized audio includes target speech and non-target speech, and the second type of noise belongs to the non-target speech.

[0063] S103: import the second recognition audio into a pre-built recognition model, and process the target speech in the second recognition audio to obtain a corresponding text.

[0064] Taking the live broadcast application scenario as an example, the audio to be identified can be the audio collected by the audio collection device during the live broadcast of the anchor. The audio information contains the anchor's voice and may also contain the ambient sound of the anchor's environment. For example, if the anchor is broadcasting live on the street, it may contain more background sound. In addition, the audio to be identified may also include the audio emitted by the live broadcast provider 100 itself. For example, in the process of live broadcasting a game, in addition to the anchor's voice, it also includes the audio of the game emitted by the device, such as the sound effects in the game, the voices emitted by the characters in the game, etc.

[0065] In live webcasting applications, the audio to be recognized is often a relatively long audio segment, such as one hour or two hours. If the audio to be recognized is not processed and the subsequent speech recognition operation is performed directly, the recognition processing of the relatively long audio segment will cause a large processing burden, and the various noises therein will also interfere with the recognition.

[0066] In this embodiment, please refer to Figure 3 First, the collected audio to be recognized is imported into the pre-built first filtering model, and the first type of noise in the audio to be recognized can be filtered through the first filtering model. The first type of noise can be non-speech noise, such as the above-mentioned ambient sound, background sound, game sound effects, and silence in the audio. That is, the first filtering model can filter the first type of noise in the audio to be recognized in a targeted manner.

[0067] From the above, it can be seen that in addition to noises such as ambient sound, the audio to be recognized may also contain human voices similar to the host's voice, such as the voices of the characters in the above-mentioned game. This type of voice is also a human voice, but the host's voice needs to be recognized and converted to text in a targeted manner later, and this type of voice is not the subsequent processing object. Therefore, on the basis of the above, the first recognized audio obtained by the first filtering model is imported into the pre-constructed second filtering model to filter the second type of noise in the first recognized audio. Among them, the first recognized audio contains target voice and non-target voice, wherein the target voice is the human voice object that needs to be recognized and processed later, such as the voice of the host, and the non-target voice is a human voice, but it is not a human voice object that needs to be recognized and processed later, such as the voice of the character in the above-mentioned game.

[0068] After the above processing, the non-speech noise and non-target speech in the audio to be recognized can be filtered out, and only the target speech in the audio to be recognized can be retained. Finally, the target speech in the second recognition audio can be processed using the pre-built recognition model to obtain the corresponding text.

[0069] The audio recognition method provided in this embodiment can not only filter out non-speech noises such as ambient sound, background sound, and device sound effects in the audio to be recognized, but also filter out non-target voices such as game characters, so as to retain only the required target voices. In this way, various meaningless noises can be effectively filtered out, and the interference of various noises on subsequent speech recognition processing can be avoided, significantly reducing misrecognition. At the same time, it can also avoid the large processing volume of the recognition conversion stage due to imperfect noise filtering in scenarios where audio processing of a longer duration is required, avoid the system's redundant calculation of meaningless audio, and avoid the problem of causing a large processing burden.

[0070] In this embodiment, the process of pre-building the first filtering model, the second filtering model and the recognition model is first introduced below.

[0071] In this embodiment, the first filtering model can be constructed using a VAD (Voice Activity Detection) algorithm. The VAD model can identify speech segments and non-speech segments from an audio segment. The VAD model can use a hard indicator detection method to filter noise, or it can use a machine learning method based on a neural network model to identify noise.

[0072] Among them, when using the hard indicator detection method, voice segments and non-voice segments can be identified by detecting volume, duration, etc. When using this detection method, the VAD model does not need to be trained in advance, and can directly use the detection-related hard indicators to achieve recognition in the application stage.

[0073] When using a method based on a neural network model, a neural network model can be pre-built, including but not limited to a radial basis network model, a recurrent neural network model, a long short-term memory model, etc., and audio training samples containing non-speech noise can be collected, such as the audio information of the anchor side in the live broadcast scene generated in the historical stage as training samples. The non-speech noise is labeled, and the constructed neural network model is trained using the training samples until the training meets the set requirements, and a VAD model obtained by training based on the neural network model is obtained.

[0074] In this embodiment, when the VAD model adopts a hard indicator detection method, non-speech noise in the audio can be detected by volume, duration, etc. When the VAD model is trained based on a neural network model, it can effectively identify speech segments and non-speech segments in the audio. Moreover, it is more sensitive to low-volume audio and can identify low-volume audio in the audio. That is, the VAD model obtained by combining two different methods can effectively detect and identify low-volume background sounds, ambient sounds, silence (no volume), and other non-speech sounds in the audio.

[0075] On the basis of the above, the present embodiment can also pre-train to obtain a second filtering model and a recognition model, wherein the second filtering model and the recognition model can be a joint model constructed by ASR (Automatic Speech Recognition, automatic speech recognition technology).

[0076] For the second filtering model in the ASR system, a neural network model can be established in advance, and a plurality of training samples containing target speech and non-target speech can be used for training to obtain the established neural network model, wherein the training samples used can be the results output by the above-mentioned first filtering model. The first filtering model can only recognize and filter out non-speech segments in the training samples. That is, the training samples used for training the second filtering model contain target speech and non-target speech. The non-target speech in the training samples, that is, the above-mentioned second type of noise, can be labeled. The established neural network model is trained using the training samples carrying the label marks, and then a second filtering model that meets the requirements is obtained. Of course, other collected training samples containing non-target speech can also be used. During the training process of the second filtering model, the model can recognize the non-target speech in the training samples and filter the non-target speech.

[0077] Among them, since the non-target speech is often a speech segment lasting a certain length of time, in order to be able to mark the non-target speech in the form of speech segments, for the second type of noise in the training samples, the second type of noise in each training sample is marked with a start symbol and an end symbol, and the second type of noise is composed of multiple audio frames of the non-target speech. The second filtering model can be trained by decoding the start symbol and the end symbol of the second type of noise in each training sample.

[0078] For a continuous second type of noise, the start symbol is marked at the start audio frame of the second type of noise, and the end symbol is marked at the end audio frame of the second type of noise. For example, the start symbol can be marked as " <s> ", the end character can be marked as "< / s> ". Both the start and end characters are control characters that can be used to guide the model to decode correctly.

[0079] During the training phase, the second filtering model can perform feature recognition and decoding on each audio frame for the input training sample. If the decoded start symbol indicates that the corresponding audio frame is a non-target speech, if the subsequent audio frames continue to be non-target speech until the decoded end symbol, it means that this section is all non-target speech, which can be determined as the second type of noise and filtered. If the second filtering model is in the training phase, during the decoding process of the input training sample, if the decoded start symbol is obtained, but the decoded end symbol is not obtained, that is, the decoded target speech is obtained, indicating that in the audio segment corresponding to the start symbol and the end symbol, there are audio frames that contain both non-target speech and non-target speech. In this case, the duration of the pure non-target speech can be counted. If the duration reaches a certain length, it can be determined that the pure non-target speech is the second type of noise, and the second type of noise is filtered.

[0080] In addition, the second filtering model can also identify the target speech and non-target speech by learning the semantic context information of the audio in the audio training sample. Since the target speech is a speech segment such as the host's voice, the target speech generally has coherence and logic between the contexts in semantic expression. Non-target speech generally does not have obvious coherence in semantic expression. Therefore, by learning the context information of the audio, it can be used in the subsequent practical application stage, and the semantic context information of the audio to be identified can be combined with decoding to identify the target speech and non-target speech.

[0081] In this embodiment, the training samples used can be samples collected containing multiple different types of noise, such as training samples mixed with the voice of game characters, training samples mixed with the voice when music is playing, etc. These different voice noises have certain differences in audio characteristics. In this embodiment, the second type of noise in each training sample is also marked with a noise label, and the noise label can identify the type of the second type of noise.

[0082] In this way, during the training stage of the second filtering model, the characteristic information of training samples of different noise types can also be learned based on the noise labels marked by the training samples. Later in the actual application stage, the second filtering model can not only filter out the second type of noise in the audio, but also identify the specific noise type of the second type of noise, which can be used for operation and management of the live broadcast platform.

[0083] On the basis of the above, during the training phase of the recognition model in the ASR system, training samples containing various types of noise can be collected. Among them, the training samples contain target speech in addition to various types of noise, and the purpose of training the recognition model is to convert the target speech into corresponding text. In this embodiment, the target speech in the training sample can be labeled, and the label can be specific text information. The recognition model is trained using the training samples labeled with text information, so that the recognition model can learn the mapping relationship between the target speech and the text.

[0084] The above process is a process of pre-training the first filtering model, the second filtering model and the recognition model. After the model that meets the requirements is obtained through training, it can be applied to the recognition processing of the audio to be recognized in the actual application stage.

[0085] See also Figure 4 Corresponding to the above implementation, when using the VAD model and ASR system to recognize the audio to be recognized, first, the audio to be recognized is input into the VAD model, the first type of noise therein is truncated and filtered out to obtain the first recognized audio, the first recognized audio is input into the ASR system, the second type of noise is first truncated and filtered out in the ASR system to obtain the second recognized audio, and then speech recognition is performed on the second recognized audio to output the corresponding text.

[0086] It can be seen from the above that the first filtering model can realize noise detection based on a hard indicator detection method, and can also perform noise detection based on a model obtained by training a neural network model.

[0087] See also Figure 5 As a possible implementation method, when the first filtering model is used to filter the first type of noise, it can be implemented in the following way:

[0088] S1011, using a pre-built first filtering model to perform volume detection on each audio frame included in the audio to be recognized.

[0089] S1012: Determine the detected audio frames whose volume is lower than a preset volume as the first type of noise, and intercept and filter the audio frames determined as the first type of noise from the audio to be recognized to obtain a first recognized audio.

[0090] For noises such as ambient sound and background sound, the volume is often low, and the VAD model is sensitive to low-volume audio and can detect low-volume audio. Therefore, a first filtering model in the form of a pre-established VAD model can be used to detect audio frames with a volume lower than a preset volume in the audio to be identified as the first type of noise and filter them. In addition, the VAD model can also identify speech segments and non-speech segments. Therefore, in addition to detecting audio with a lower volume, non-speech segments can also be identified and used as the first type of noise.

[0091] When filtering the first type of noise, assuming that from 5s to 10s is detected as the first type of noise, the audio to be recognized can be truncated from before the audio frame of the 5s and after the audio frame of the 10s, the truncated audio segment from the 5s to the 10s is removed from the audio to be recognized, and the truncated parts are spliced ​​to obtain the first recognized audio with the first type of noise filtered out.

[0092] On this basis, the pre-trained second filtering model can be used to decode the first recognition audio, thereby filtering out the second type of noise. Figure 6 In this embodiment, this step can be implemented in the following way:

[0093] S1021A, using the pre-constructed second filtering model, decoding starts from the first audio frame in the first recognized audio to obtain corresponding audio features.

[0094] S1022A, when the audio feature representation of the decoded audio frame is non-target speech, and there is an audio frame of a set frame whose audio feature representation is non-target speech starting from the audio frame, determine the audio frame of the set frame as the second type of noise, wherein the set frame exceeds a preset threshold.

[0095] S1023A, intercepting and filtering the audio frame of the set frame from the first recognized audio, and continuing to process subsequent audio frames until the processing of the first recognized audio is completed.

[0096] As can be seen from the above, when the second filtering model is trained in advance, the second filtering model can learn the feature information of the non-target speech and the feature information of the target speech. For the first recognition audio input into the second filtering model, the second filtering model can decode from the first audio frame and obtain the audio features corresponding to each audio frame one by one.

[0097] The second filtering model can determine whether the audio frame is a non-target speech or a target speech based on the audio features obtained after decoding. In this embodiment, it is considered that if the duration of the non-target speech is very short, it will not cause much interference to the subsequent speech recognition, and if the truncation process is performed every time a non-target speech is encountered, it will also cause a processing burden of filtering and truncation. Therefore, in this embodiment, when the audio frame is decoded as a non-target speech and the audio frames of the set frame starting from the audio frame are all non-target speech, it is determined that the audio frame of the set frame is the second type of noise, that is, the noise that needs to be filtered out.

[0098] The set frame is set to exceed a preset threshold, for example, the preset threshold may be 5 frames, and the set frame may be 6 frames, 7 frames, 8 frames, etc. without limitation.

[0099] For example, the preset threshold is set to 5 frames. If the 5th frame of the audio in the first recognition audio is determined to be a non-target speech, and the 6th to 10th frames are all non-target speech, then the 5th to 10th frames can be determined as the second type of noise, and the second type of noise in this section is filtered. If the 5th to 9th frames are determined to be non-target speech, but the 10th frame is the target speech, then the 5th to 9th frames are not determined to be the second type of noise and are still retained in the first recognition audio.

[0100] It should be noted that the audio frames filtered out by the second filtering model are audio frames containing only non-target speech. If an audio frame is characterized as both target speech and non-target speech, then the audio frame should be determined as target speech. During filtering, if frames 5 to 15 are non-target speech, for example, there is continuous voice information of the game character from frames 5 to 15, but frames 11 to 15 are characterized as target speech at the same time, indicating that there are both voices like the host and voices like the characters in the game in frames 11 to 15, then frames 5 to 10 can be intercepted and determined as the second type of noise and filtered out.

[0101] Through the above method, it is possible to avoid excessive processing burden in the filtering stage without causing significant interference to subsequent audio recognition.

[0102] In this embodiment, it can be seen from the above that the second filtering model can recognize the target speech and the non-target speech by learning the semantic context information of the audio. When the second filtering model processes the first recognized audio, the semantic context information of the decoded audio frame will be obtained.

[0103] After the second type of noise in the first recognized audio is determined and intercepted and filtered, generally, there may be a discontinuity in the continuity of the semantic context from the interception point onwards. In order to avoid the influence of the previous semantic context on the decoding process of the subsequent audio, the semantic context information stored in the second filter model can be initialized, that is, the decoding state of the second filter model is reset, and then the processing of the subsequent audio frames is continued until the processing of the first recognized audio is completed.

[0104] In this embodiment, based on the above method, the non-target speech in the first recognition audio can be filtered out by the second filtering model. However, since there are short pauses between characters and words in Chinese speech expression, the audio frames of the target speech have audio frames corresponding to the short pauses, and such audio frames are actually silent. Therefore, in order to further reduce the processing workload of subsequent speech recognition, please refer to Figure 7 In a possible implementation, when the second filtering model is used for filtering, it can be implemented in the following manner:

[0105] S1021B: Use a pre-constructed second filtering model to identify audio frames characterized as target speech and audio frames characterized as non-target speech in the first recognized audio.

[0106] S1022B, determining a second type of noise from the recognized audio frames of the non-target speech, and determining a blank frame from the recognized audio frames of the target speech.

[0107] S1023B, intercept and filter out the second type of noise and the blank frame from the first recognition audio.

[0108] In this embodiment, on the basis of determining the second type of noise in the first recognition audio using the above method, blank frames in the audio frames of the target speech can also be identified, so that the second type of noise and blank frames are filtered out from the first recognition audio.

[0109] Among them, a CTC algorithm, an end-to-end speech recognition method, can be added to the second filtering model. The CTC algorithm can establish a sequence mapping relationship between audio and text, and the classification label of the audio frame can be output frame by frame through the loss function of the CTC algorithm, so as to determine the blank frames therein, and the blank frames can be intercepted and filtered out from the audio segment.

[0110] In this embodiment, combined with the above-mentioned blank frame filtering method, silence missed in the first filtering model filtering stage can be detected, further reducing the processing load of subsequent speech recognition.

[0111] After obtaining the second recognized audio in the above manner, the recognition model can be used to process the target speech in the second recognized audio to obtain the corresponding text. As can be seen from the above, the second recognized audio may contain some non-target speech but not determined as the second type of noise audio frames, that is, non-target speech with a very short duration. In this case, please refer to Figure 8 , when performing text conversion, it can be achieved in the following ways:

[0112] S1031: When there are audio frames in the second recognized audio whose audio features are characterized as non-target speech and the number of consecutive frames is lower than a preset threshold, determine the audio frames in the second recognized audio whose audio features are characterized as target speech.

[0113] S1032, obtaining the text corresponding to the audio frame whose determined audio feature representation is the target speech.

[0114] In this embodiment, when the second recognition audio contains audio frames of the target speech and non-target speech, the target speech is determined. When converting to text, only the target speech is converted, and the non-target speech is not output as text, so that some unnecessary workload can be avoided and interference caused by meaningless semantics in the output text can be avoided.

[0115] The audio recognition method provided in this embodiment can filter non-speech noise through the first filtering model based on the VAD model, and utilize the VAD model's detection advantages for hard indicators such as volume and processing advantages in semantic recognition to hierarchically truncate and process the audio, effectively filtering out non-speech noise in the audio. In addition, the second filtering model in the ASR system is combined to filter non-target speech.

[0116] By using the VAD model's sensitivity to low volume to filter out some background sounds, ambient sounds, and silence, we can make up for the ASR system's insensitivity to low volume and avoid using the ASR system to process extremely low volume background sounds, which can affect the recognition effect of the entire system. In addition, the VAD model is relatively small in size, which can avoid the problem of relying solely on the ASR system to process audio and requiring a large amount of calculation.

[0117] For the speech recognition stage of the recognition model, we can use noise-doped training samples that are more in line with actual application scenarios. Compared with the existing method of using training samples that only contain human voices for training, the trained recognition model can be more able to learn the audio features of real application scenarios, improve the accuracy of speech recognition conversion, and can be widely used in the recognition and processing of noisy and long-duration audio such as in live broadcast scenarios.

[0118] See also Fig. 9 The embodiment of the present application also provides an electronic device. The electronic device can be used as a live broadcast device, for example, it can be a terminal device used by the anchor during live broadcast (such as the above-mentioned live broadcast provider 100), or it can be a server (such as the above-mentioned live broadcast server 200) that is connected to the terminal device used by the anchor during live broadcast.

[0119] In detail, the electronic device may include a memory 410, a processor 420 and an audio recognition device 430. The memory 410 and the processor 420 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, they may be electrically connected to each other via one or more communication buses or signal lines. The audio recognition device 430 includes at least one software function module that can be stored in the memory 410 in the form of software or firmware. The processor 420 is used to execute an executable computer program stored in the memory 410, for example, the software function module and computer program included in the audio recognition device 430, to achieve filtering of various types of noise in the audio and speech recognition processing.

[0120] Among them, the memory 410 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0121] The memory 410 is used to store programs, and the processor 420 executes the programs after receiving the execution instruction. The processor 420 may be an integrated circuit chip with signal processing capabilities. For example, it may be a central processing unit (CPU), a network processor (NP), a system on chip (SoC), a digital signal processor (DSP), etc., to implement or execute the methods and steps disclosed in the embodiments of the present application.

[0122] Understandably, Fig. 9 The structure shown is for illustration only, and the electronic device may also include Fig. 9More or fewer components as shown, or with Fig. 9 The different configurations shown, for example, may also include a communication unit for exchanging information with other live broadcast devices. Fig. 9 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0123] See also Fig.10 , is a functional module block diagram of an audio recognition device 430 provided in an embodiment of the present application. The audio recognition device 430 includes a first filtering module 431 , a second filtering module 432 and a processing module 433 .

[0124] The first filtering module 431 is used to filter the first type of noise in the audio to be recognized by using a pre-built first filtering model to obtain a first recognized audio, where the first type of noise is non-speech noise.

[0125] In this embodiment, the first filtering module 431 may be used to perform Figure 2 As shown in step S101 , the relevant contents about the first filtering module 431 can refer to the above description of step S101 .

[0126] The second filtering module 432 is used to filter the second type of noise in the first recognized audio using a pre-built second filtering model to obtain a second recognized audio, wherein the first recognized audio contains target speech and non-target speech, and the second type of noise belongs to the non-target speech.

[0127] In this embodiment, the second filtering module 432 may be used to perform Figure 2 As shown in step S102, the relevant contents of the second filtering module 432 can refer to the above description of step S102.

[0128] The processing module 433 is used to import the second recognition audio into a pre-built recognition model, and process the target speech in the second recognition audio to obtain the corresponding text.

[0129] In this embodiment, the processing module 433 can be used to execute Figure 2 As shown in step S103, the relevant contents of the processing module 433 can refer to the above description of step S103.

[0130] In a possible implementation, the first filtering module 431 may be specifically used for:

[0131] Using a pre-built first filtering model to perform volume detection on each audio frame included in the audio to be recognized;

[0132] The detected audio frames whose volume is lower than a preset volume are determined as first-type noise, and the audio frames determined as first-type noise are intercepted and filtered out from the audio to be recognized to obtain first recognized audio.

[0133] In a possible implementation, the second filtering module 432 may be specifically used to:

[0134] Using the pre-built second filtering model, decoding is started from the first audio frame in the first recognized audio to obtain corresponding audio features;

[0135] When the audio feature of the decoded audio frame is characterized as a non-target speech, and there is an audio frame of a set frame whose audio feature is characterized as a non-target speech starting from the audio frame, determining the audio frame of the set frame as the second type of noise, wherein the set frame exceeds a preset threshold;

[0136] The audio frame of the set frame is intercepted and filtered out from the first recognized audio, and the processing of subsequent audio frames is continued until the processing of the first recognized audio is completed.

[0137] In a possible implementation, the second filtering model stores semantic context information of the decoded audio frame. The second filtering module 432 may also be used to:

[0138] Initialize the semantic context information stored in the second filtering model.

[0139] In a possible implementation, the processing module 433 may be specifically used to:

[0140] When there are audio frames in the second recognized audio whose audio features are characterized as non-target speech and the number of consecutive frames is lower than a preset threshold, determining the audio frames in the second recognized audio whose audio features are characterized as target speech;

[0141] The determined audio feature representation is obtained as text corresponding to the audio frame of the target speech.

[0142] In a possible implementation, the second filtering module 432 may be used to:

[0143] Using a pre-built second filtering model, identify audio frames characterized as target speech and audio frames of non-target speech in the first recognition audio;

[0144] Determine a second type of noise from the recognized audio frame of the non-target speech, and determine a blank frame from the recognized audio frame of the target speech;

[0145] The second type of noise and the blank frame are intercepted and filtered out from the first recognition audio.

[0146] In a possible implementation, the second filtering model is pre-trained based on a plurality of training samples including target speech and non-target speech;

[0147] The second type of noise in each of the training samples is marked with a start symbol and an end symbol, the second type of noise is composed of multiple audio frames of non-target speech, and the second filtering model is trained by decoding the start symbol and the end symbol of the second type of noise in each of the training samples.

[0148] In a possible implementation manner, the second type of noise in each of the training samples is further marked with a noise label, and the noise label identifies the type of the second type of noise.

[0149] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0150] In an embodiment of the present application, corresponding to the above-mentioned audio recognition method, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is run, each step of the above-mentioned audio recognition method is executed.

[0151] Among them, the steps executed when the aforementioned computer program is running will not be described one by one here, and reference can be made to the explanation of the audio recognition method in the previous text.

[0152] In summary, the audio recognition method, device, electronic device and readable storage medium provided in the embodiments of the present application first use a pre-constructed first filtering model to filter the first type of noise in the audio to be recognized to obtain a first recognized audio, where the first type of noise is non-speech noise. Then, a pre-constructed second filtering model is used to filter the second type of noise in the first recognized audio to obtain a second recognized audio, where the first recognized audio contains target speech and non-target speech, and the second type of noise belongs to non-target speech. Finally, the second recognized audio is imported into the pre-constructed recognition model, and the target speech in the second recognized audio is processed to obtain the corresponding text. This solution pre-filters non-speech noise and noise belonging to non-target speech, reduces the processing pressure of speech recognition and eliminates the interference of various types of noise, and has a better effect on audio recognition processing of a longer duration and mixed with noise.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. An audio recognition method, characterized in that: The method comprises: Using a pre-built first filtering model to filter a first type of noise in the audio to be recognized to obtain a first recognized audio, wherein the first type of noise is non-speech noise; Using a pre-constructed second filtering model to filter the second type of noise in the first recognized audio to obtain a second recognized audio, the first recognized audio includes a target voice and a non-target voice, the second type of noise belongs to the non-target voice, and in the process of processing the first recognized audio by the second filtering model, the semantic context information of the decoded audio frame is obtained and saved, and each time the second type of noise in the first recognized audio is determined and intercepted and filtered, the saved semantic context information is initialized, and then the processing of subsequent audio frames is continued until the processing of the first recognized audio is completed; The second recognition audio is imported into a pre-built recognition model, and the target speech in the second recognition audio is processed to obtain the corresponding text.

2. The audio recognition method according to claim 1, characterized in that: The step of filtering the first type of noise in the audio to be recognized by using the pre-built first filtering model to obtain the first recognized audio includes: Using a pre-built first filtering model to perform volume detection on each audio frame included in the audio to be recognized; The detected audio frames whose volume is lower than a preset volume are determined as first-type noise, and the audio frames determined as first-type noise are intercepted and filtered out from the audio to be recognized to obtain first recognized audio.

3. The audio recognition method according to claim 1, characterized in that: The step of filtering the second type of noise in the first recognized audio using the pre-built second filtering model to obtain the second recognized audio includes: Using the pre-built second filtering model, decoding is started from the first audio frame in the first recognized audio to obtain corresponding audio features; When the audio feature of the decoded audio frame is characterized as a non-target speech, and there is an audio frame of a set frame whose audio feature is characterized as a non-target speech starting from the audio frame, determining the audio frame of the set frame as the second type of noise, wherein the set frame exceeds a preset threshold; The audio frame of the set frame is intercepted and filtered out from the first recognized audio, and the processing of subsequent audio frames is continued until the processing of the first recognized audio is completed.

4. The audio recognition method according to claim 3, characterized in that: The step of importing the second recognized audio into a pre-built recognition model and processing the target speech in the second recognized audio to obtain the corresponding text includes: When there are audio frames in the second recognized audio whose audio features are characterized as non-target speech and the number of consecutive frames is lower than a preset threshold, determining the audio frames in the second recognized audio whose audio features are characterized as target speech; The determined audio feature representation is obtained as text corresponding to the audio frame of the target speech.

5. The audio recognition method according to claim 1, characterized in that: The step of filtering the second type of noise in the first recognized audio using the pre-built second filtering model to obtain the second recognized audio includes: Using a pre-built second filtering model, identify audio frames characterized as target speech and audio frames of non-target speech in the first recognition audio; Determine a second type of noise from the recognized audio frame of the non-target speech, and determine a blank frame from the recognized audio frame of the target speech; The second type of noise and the blank frame are intercepted and filtered out from the first recognition audio.

6. The audio recognition method according to any one of claims 1 to 5, characterized in that: The second filtering model is obtained by pre-training based on a plurality of training samples including target speech and non-target speech; The second type of noise in each of the training samples is marked with a start symbol and an end symbol, the second type of noise is composed of multiple audio frames of non-target speech, and the second filtering model is trained by decoding the start symbol and the end symbol of the second type of noise in each of the training samples.

7. The audio recognition method according to claim 6, characterized in that: The second type of noise in each of the training samples is also marked with a noise label, and the noise label identifies the type of the second type of noise.

8. An audio recognition device, characterized in that: For implementing the audio recognition method according to any one of claims 1 to 7, the device comprises: A first filtering module, configured to filter a first type of noise in the audio to be recognized by using a pre-built first filtering model to obtain a first recognized audio, wherein the first type of noise is non-speech noise; A second filtering module, configured to filter the second type of noise in the first recognized audio using a pre-constructed second filtering model to obtain a second recognized audio, wherein the first recognized audio includes target speech and non-target speech, and the second type of noise belongs to the non-target speech; A processing module is used to import the second recognition audio into a pre-built recognition model and process the target speech in the second recognition audio to obtain the corresponding text.

9. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed, the method steps described in any one of claims 1 to 7 are implemented.

Images