Effective speech recognition method and device

Abstract

The invention provides an effective speech recognition method and device, and the method comprises the steps: extracting the audio features of to-be-recognized audio data based on an effective speech recognition model, and determining effective speech data from the to-be-recognized audio data through the audio features of the to-be-recognized audio data; the effective speech recognition model minimizes the difference between the effective predicted speech and the effective speech tag and minimizes the distance between the audio feature of the sample audio data and the audio feature of the noise-added sample audio data. The distance between the audio feature of the sample audio data and the audio feature of the pure noise data is maximized as a training target, and the effective prediction voice is obtained by performing effective voice recognition on the sample audio data through the effective voice recognition model. According to the invention, effective speech recognition can be accurately carried out on the to-be-recognized audio data in a scene with relatively small speech signal-to-noise ratio and relatively large background noise, and the effective speech recognition precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of audio signal processing, and in particular to an effective speech recognition method and device. Background Art

[0002] In the field of audio signal processing technology, it is often necessary to use VAD (Voice Activity Detection) technology to identify voice segments in audio data, and perform speech recognition, semantic recognition and other processing on the identified voice segments according to different audio data processing requirements.

[0003] At present, the audio features in the audio data are mostly extracted through neural networks, and the speech segments and non-speech segments are distinguished from the audio data based on the audio features. However, this method has poor effective speech recognition effect for audio data with a small speech signal-to-noise ratio. Summary of the invention

[0004] The present invention provides an effective speech recognition method and device to solve the defects in the prior art.

[0005] The present invention provides an effective speech recognition method, comprising the following steps: Determining audio data to be recognized; Extracting audio features of the audio data to be recognized based on the effective speech recognition model, and applying the audio features of the audio data to be recognized to determine effective speech data from the audio data to be recognized; The effective speech recognition model has the training objectives of minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and maximizing the distance between the audio features of the sample audio data and the audio features of pure noise data. The effective predicted speech is obtained by the effective speech recognition model through effective speech recognition of the sample audio data.

[0006] According to an effective speech recognition method provided by the present invention, the training step of the effective speech recognition model includes: Based on the initial model, extracting audio features of the sample audio data, audio features of the noisy sample audio data, and audio features of the pure noise data; Based on the initial model, applying the audio features of the sample audio data, determining the effective predicted speech; Determine a training loss value based on the difference between the effective predicted speech and the effective speech label, the distance between the audio feature of the sample audio data and the audio feature of the noisy sample audio data, and the distance between the audio feature of the sample audio data and the audio feature of the pure noise data; Based on the training loss value, the parameters of the initial model are updated to obtain the effective speech recognition model.

[0007] According to an effective speech recognition method provided by the present invention, the training loss value is determined based on the difference between the effective predicted speech and the effective speech label, the distance between the audio feature of the sample audio data and the audio feature of the noisy sample audio data, and the distance between the audio feature of the sample audio data and the audio feature of the pure noise data, including: Determining a classification loss value based on a difference between the valid predicted speech and the valid speech label; Determining an intra-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of the noisy sample audio data; Determining an inter-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of pure noise data; Based on preset weights, the classification loss value, the intra-class loss value, and the inter-class loss value are weightedly added to obtain the training loss value.

[0008] According to an effective speech recognition method provided by the present invention, the effective speech data is determined from the audio data to be recognized, and then the method further includes: Extracting voiceprint features of each segment in the valid voice data; Based on the similarity between the voiceprint features of each segment and the noise garbage voiceprint features, and the similarity between the voiceprint features of each segment and the distortion signal garbage voiceprint features, filtering out the garbage sound segments from the valid voice data; The noise and garbage sound voiceprint features are determined based on the voiceprint features corresponding to a plurality of sample noise and garbage sounds, and the distortion signal and garbage sound voiceprint features are determined based on the voiceprint features corresponding to a plurality of sample distortion signal and garbage sounds.

[0009] According to an effective speech recognition method provided by the present invention, based on the similarity between the voiceprint features of each segment and the noise garbage voiceprint features, and the similarity between the voiceprint features of each segment and the distorted signal garbage voiceprint features, the garbage sound segments are filtered out from the effective speech data, including: If the similarity between the voiceprint features of any segment and the noise garbage voiceprint features is greater than or equal to a threshold, and / or the similarity between the voiceprint features of any segment and the distorted signal garbage voiceprint features is greater than or equal to the threshold, then any segment is taken as the garbage segment and the garbage segment is filtered out from the valid voice data.

[0010] According to an effective speech recognition method provided by the present invention, the method of taking any of the segments as the junk sound segment further comprises: The noise and garbage voiceprint features are updated based on the voiceprint features of any one of the segments and / or the distorted signal and garbage voiceprint features are updated.

[0011] According to an effective speech recognition method provided by the present invention, the audio data to be recognized is collected in a variety of ways; The determining of valid voice data from the audio data to be recognized further comprises: Based on the duration of the valid voice data corresponding to the audio data to be recognized under each acquisition mode, the target voice data is determined from each valid voice data.

[0012] The present invention also provides an effective speech recognition device, comprising the following modules: A determination unit, used to determine the audio data to be identified; A recognition unit, configured to extract audio features of the audio data to be recognized based on a valid speech recognition model, and determine valid speech data from the audio data to be recognized by applying the audio features of the audio data to be recognized; The effective speech recognition model has the training objectives of minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and maximizing the distance between the audio features of the sample audio data and the audio features of pure noise data. The effective predicted speech is obtained by the effective speech recognition model through effective speech recognition of the sample audio data.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned effective speech recognition methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the effective speech recognition method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the effective speech recognition method as described above is implemented.

[0016] The effective speech recognition method and device provided by the present invention improve the accuracy of the effective speech recognition model in recognizing the effective speech by minimizing the difference between the effective predicted speech and the effective speech label, improve the anti-noise ability of the effective speech recognition model by minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and improve the noise discrimination ability of the effective speech recognition model by maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data. In this way, the effective speech recognition model can accurately perform effective speech recognition on the audio data to be recognized in the scenario where the speech signal-to-noise ratio is small and the background noise is large, thereby improving the effective speech recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the effective speech recognition method provided by the present invention.

[0019] Figure 2 It is a flow chart of the effective speech recognition model training method provided by the present invention.

[0020] Figure 3 It is a flow chart of the junk sound filtering method provided by the present invention.

[0021] Figure 4 It is a structural schematic diagram of an effective speech recognition device provided by the present invention.

[0022] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Audio data of the same speech content is usually collected in multiple ways, and an audio data with higher quality and the longest effective duration is selected from them, so that the audio data can be subsequently applied to tasks such as speech recognition and semantic recognition. In addition, due to the different collection methods of the multiple audio data collected above, the background noise of each audio data will be quite different. For example, in some audio data, almost only noise can be heard, in some audio data, although the background noise is not particularly large, the voice quality is seriously distorted, making it difficult to understand the speaker's voice content, and in some audio data, although the speaker's voice content can be heard clearly, it is necessary to select the audio data with the longest effective voice duration to ensure that as much voice content as possible is obtained.

[0025] If we rely on manual screening to obtain the optimal audio data from the above audio data, it will not only consume a lot of human resources, but also the manual screening speed is limited, which will lead to the problem of not being able to delete the non-optimal audio data in time and wasting a lot of storage space.

[0026] In this regard, neural networks are currently used to extract audio features from audio data, and based on the audio features, speech segments (speech) and non-speech segments (nospeech) are distinguished from audio data. However, when faced with scenes with large background noise and low speech signal-to-noise ratio, neural networks are prone to misidentifying speech and nospeech in audio data, such as misidentifying noise segments as speech, and misidentifying noisy speech segments as nospeech. The neural network here can be an endpoint detection network constructed using a DNN (Deep Neural Network) network, in which the DNN in the endpoint detection network is connected to a fully connected layer, and audio features (such as filter bank features) are input into the DNN, and the DNN transforms the audio features and then inputs them into the fully connected layer to obtain effective speech recognition results at the frame level, such as the probability of each frame being speech and the probability of nospeech.

[0027] Based on this, the present invention provides an effective speech recognition method to improve the effect of effective speech recognition in scenes with large background noise and low speech signal-to-noise ratio. Figure 1 is a flow chart of an effective speech recognition method provided by the present invention, such as Figure 1 As shown, the method includes step 110 and step 120.

[0028] Step 110: Determine the audio data to be recognized.

[0029] Here, the audio data to be recognized is the voice data that needs to be effectively recognized. The audio data to be recognized can be pure voice data or voice data with noise. The audio data to be recognized can be voice data collected in advance. The embodiment of the present invention does not limit the language type of the audio data to be recognized, such as the audio data to be recognized can be Chinese audio data, English audio data, etc. At the same time, the embodiment of the present invention does not limit the length of the audio data to be recognized, such as the audio data to be recognized can be a sentence or a paragraph, etc.

[0030] It is understandable that the audio data to be identified can be acquired in real time through a sound pickup device according to actual needs, or can be audio data downloaded from the Internet, and the embodiment of the present invention does not specifically limit this.

[0031] Step 120: extracting audio features of the audio data to be recognized based on the effective speech recognition model, and applying the audio features of the audio data to be recognized to determine effective speech data from the audio data to be recognized; The effective speech recognition model is trained with the objectives of minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data. The effective predicted speech is obtained by the effective speech recognition model performing effective speech recognition on the sample audio data.

[0032] Specifically, the audio features of the audio data to be identified can be used to describe the characteristics of the audio signal to be identified, and the audio features can distinguish different audio data. The audio features of the audio data to be identified can be any type of audio features, such as audio data spectrum, filterbank features, bottleneck features, PLP (Perceptual Linear Prediction, linear perceptual prediction) features, Mel features, log-Mel features, etc. Exemplarily, the audio features of the audio data to be identified can be specifically expressed in the form of feature vectors.

[0033] After extracting the audio features of the audio data to be recognized, the effective speech recognition model determines effective speech data from the audio data to be recognized based on the audio features. The effective speech data can be understood as a speech segment in the audio data to be recognized that can clearly express the speaker's speech content. Optionally, the effective speech recognition model can output the probability that each frame segment in the audio data to be recognized is a speech, and splice the frame segments with probabilities greater than a threshold to obtain the corresponding effective speech data.

[0034] Taking into account that traditional methods are prone to misidentify speech and nospeech in audio data when faced with scenarios with large background noise and low speech signal-to-noise ratio, the embodiments of the present invention set three training goals in the training stage of the effective speech recognition model: 1. Minimize the difference between the effective predicted speech and the effective speech label; 2. Minimize the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data; 3. Maximize the distance between the audio features of the sample audio data and the audio features of the pure noise data.

[0035] Among them, the effective predicted speech in training goal ① is the effective speech predicted by the effective speech recognition model from the sample audio data, and the effective speech label is the real effective speech corresponding to the sample audio data. The difference between the effective predicted speech and the effective speech label is used to characterize the accuracy of the effective speech recognition model in effective speech recognition based on the characteristics of the audio data itself (ie, audio characteristics). The smaller the difference between the two, the more accurately the effective speech recognition model can recognize speech from the sample audio data, that is, the stronger the effective speech recognition ability of the effective speech recognition model is. In other words, training goal ① aims to enable the effective speech recognition model to perform effective speech recognition more accurately.

[0036] The noisy sample audio data in training objective ② refers to the audio data obtained after the sample audio data is subjected to noise processing. For example, one or more pure noises can be randomly added to the sample audio data according to business requirements to obtain the noisy sample audio data. The distance between the audio features of the sample audio data and the audio features of the noisy sample audio data is used to characterize the effective speech recognition ability of the effective speech recognition model in a noisy scene. The smaller the distance between the two, the more accurate the effective speech recognition model can perform effective speech recognition in a noisy scene, that is, training objective ②. Considering that in actual applications, audio data is often accompanied by various noises, such as environmental noise, equipment noise, etc., training objective ② aims to minimize the distance between the audio features before and after the noise, so that the effective speech recognition model can still maintain a high recognition accuracy for the noisy sample audio data, that is, to improve the anti-noise ability of the effective speech recognition model, so that the effective speech recognition model can better adapt to the noisy data and improve the accuracy of effective speech recognition under noisy conditions.

[0037] The distance between the audio features of the sample audio data and the audio features of the pure noise data in the training objective ③ is used to characterize the ability of the effective speech recognition model to distinguish between noise data and speech data. The larger the distance between the two, the more accurately the effective speech recognition model can distinguish between noise data and speech data in a noisy scenario. That is, the training objective ③ aims to maximize the distance between the audio features of the sample audio data and the audio features of the pure noise data, so that the effective speech recognition model can accurately distinguish between speech signals and noise signals, thereby avoiding the interference of noise signals on the effective speech recognition results, that is, improving the noise distinction ability of the effective speech recognition model. The sample audio data here can be understood as pure speech data.

[0038] It can be seen that the effective speech recognition method provided by the embodiment of the present invention improves the accuracy of the effective speech recognition model in identifying effective speech by minimizing the difference between the effective predicted speech and the effective speech label, improves the noise resistance of the effective speech recognition model by minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and improves the noise distinction ability of the effective speech recognition model by maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data. As a result, the effective speech recognition model can accurately perform effective speech recognition on the audio data to be recognized in scenarios where the speech signal-to-noise ratio is relatively small and the background noise is large, thereby improving the effective speech recognition accuracy.

[0039] Based on the above embodiments, Figure 2 is a flow chart of an effective speech recognition model training method provided by the present invention, such as Figure 2 As shown, the training steps for an effective speech recognition model include: Step 210: extracting audio features of sample audio data, audio features of noisy sample audio data, and audio features of pure noise data based on the initial model.

[0040] Specifically, the initial model can be constructed based on a pre-trained network and a DNN network. The pre-trained network can include a CPC neural network, a wav2vec2.0 neural network, etc. Taking the CPC network as an example, the pre-trained network can use a non-linear coding layer (such as a 5-layer CNN+ReLU) to downsample and encode the input sample audio data, the noisy sample audio data, and the pure noise data to obtain latent variables, and then use an autoregressive model (such as a layer of GRU RNN) to encode the latent variables of the historical moments to obtain context features, that is, to obtain the corresponding audio features.

[0041] The pre-trained network can be trained based on historical audio data and audio feature labels corresponding to the historical audio data. During the training process, the pre-trained network extracts corresponding historical audio features from the historical audio data, determines the contrastive learning loss based on the historical audio features and the audio feature labels, and updates the network parameters of the pre-trained network based on the contrastive learning loss. InfoNCE (Information Noise Contrastive Estimation) can be used to determine the above contrastive learning loss, and other methods can also be used to determine the contrastive learning loss, which is not specifically limited in the embodiments of the present invention.

[0042] After obtaining the trained pre-trained network, the pre-trained network is connected to the DNN network to obtain the above-mentioned initial model, and then the sample audio data, the noisy sample audio data, and the pure noise data are respectively input into the initial model, and the pre-trained network of the initial model extracts the audio features of the sample audio data, the audio features of the noisy sample audio data, and the audio features of the pure noise data.

[0043] Step 220: Based on the initial model, apply the audio features of the sample audio data to determine the effective predicted speech.

[0044] Specifically, after the audio features of the sample audio data are obtained, the audio features of the sample audio data extracted in step 210 are input into the DNN network of the initial model, and the DNN network determines the valid speech in the sample audio data, that is, obtains the valid predicted speech. Optionally, the DNN network can output the probability that each frame segment in the sample audio data is a valid speech, and splice the frame segments with probabilities greater than a threshold to obtain the valid predicted speech.

[0045] Step 230: Determine the training loss value based on the difference between the effective predicted speech and the effective speech label, the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and the distance between the audio features of the sample audio data and the audio features of the pure noise data.

[0046] Step 240: Based on the training loss value, update the parameters of the initial model to obtain an effective speech recognition model.

[0047] Specifically, the difference between the effective predicted speech and the effective speech label is used to characterize the accuracy of the effective speech recognition model in effective speech recognition based on the characteristics of the audio data itself (i.e., audio features). The smaller the difference between the two, the more accurately the effective speech recognition model can recognize speech from the sample audio data, that is, the stronger the effective speech recognition ability of the effective speech recognition model.

[0048] The distance between the audio features of the sample audio data and the audio features of the sample audio data after noise is added is used to characterize the effective speech recognition ability of the effective speech recognition model in a noisy scene. The smaller the distance between the two, the more accurate the effective speech recognition model can be in a noisy scene. That is, minimizing the distance between the audio features of the sample audio data and the audio features of the sample audio data after noise is added can improve the noise resistance of the effective speech recognition model, so that the effective speech recognition model can better adapt to the noise data and improve the accuracy of effective speech recognition under noisy conditions.

[0049] The distance between the audio features of the sample audio data and the audio features of the pure noise data is used to characterize the ability of the effective speech recognition model to distinguish between noise data and speech data. The larger the distance between the two, the more accurately the effective speech recognition model can distinguish between noise data and speech data in a noisy scenario. That is, maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data can enable the effective speech recognition model to accurately distinguish between speech signals and noise signals, thereby avoiding the interference of noise signals on the effective speech recognition results, that is, improving the noise distinction ability of the effective speech recognition model.

[0050] The embodiment of the present invention determines the training loss value by combining the difference between the effective predicted speech and the effective speech label, the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and the distance between the audio features of the sample audio data and the audio features of the pure noise data. Therefore, after the parameters of the initial model are updated based on the training loss value, the obtained effective speech recognition model can accurately perform effective speech recognition on the audio data to be recognized in scenarios where the speech signal-to-noise ratio is relatively small and the background noise is relatively large, thereby improving the accuracy of effective speech recognition.

[0051] It is understandable that when updating the parameters of the initial model based on the training loss value, the convergence condition can be reached when the training loss value reaches stability (such as less than a threshold) or reaches the maximum number of iterations, and an effective speech recognition model can be obtained.

[0052] Based on any of the above embodiments, determining a training loss value based on a difference between the effective predicted speech and the effective speech label, a distance between an audio feature of the sample audio data and an audio feature of the noisy sample audio data, and a distance between an audio feature of the sample audio data and an audio feature of pure noise data, comprises: Determining a classification loss value based on the difference between the valid predicted speech and the valid speech label; Determining an intra-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of the noisy sample audio data; Determining an inter-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of pure noise data; Based on the preset weights, the classification loss value, intra-class loss value, and inter-class loss value are weighted added to obtain the training loss value.

[0053] Specifically, the difference between the effective predicted speech and the effective speech label is used to characterize the accuracy of the effective speech recognition model based on the characteristics of the audio data itself (i.e., the audio characteristics). The smaller the difference between the two, the smaller the corresponding classification loss value, indicating that the effective speech recognition model is more able to accurately recognize speech from the sample audio data, that is, the effective speech recognition ability of the effective speech recognition model is stronger. Optionally, the classification loss value can be determined using a binary cross entropy loss function (BCELoss).

[0054] The distance between the audio features of the sample audio data and the audio features of the sample audio data after adding noise is used to characterize the effective speech recognition ability of the effective speech recognition model in a noisy scene. The smaller the distance between the two, the smaller the corresponding intra-class loss value, indicating that the effective speech recognition model can more accurately perform effective speech recognition in a noisy scene. Optionally, the intra-class loss value can be determined using the mean squared error loss function (MSELoss).

[0055] The distance between the audio features of the sample audio data and the audio features of the pure noise data is used to characterize the ability of the effective speech recognition model to distinguish between noise data and speech data. The larger the distance between the two, the smaller the corresponding inter-class loss value, indicating that the effective speech recognition model can more accurately distinguish between noise data and speech data in a noisy scenario. Optionally, the inter-class loss value can be determined using a triplet loss function (TripletLoss).

[0056] During training, based on the preset weights, the classification loss value, intra-class loss value and inter-class loss value are weighted added to obtain the training loss value. That is, the training loss value simultaneously takes into account the model's effective speech recognition ability, noise resistance ability and noise discrimination ability, so that the trained effective speech recognition model can accurately perform effective speech recognition on the recognized audio data in scenarios with relatively small speech signal-to-noise ratio and large background noise, thereby improving the effective speech recognition accuracy.

[0057] Optionally, the training loss value can be determined based on the following formula: in, represents the training loss value, represents the classification loss value, represents the intra-class loss value, represents the inter-class loss value, for The corresponding weight, for The corresponding weight, and It can be set according to the actual situation, such as is 0.02, is 0.01.

[0058] Although the above-mentioned effective speech recognition model can accurately perform effective speech recognition on the audio data to be recognized and improve the accuracy of effective speech recognition in the scenario where the speech signal-to-noise ratio is small and the background noise is large, the effective speech recognition model may still recognize these untrained data as effective speech when facing untrained data (such as some pure noise data or audio data in which the speaker's speech content cannot be heard due to severe speech distortion). Based on this, after determining the effective speech data from the audio data to be recognized, the embodiment of the present invention further identifies junk sound segments (including pure noise segments and severely distorted speech segments) in the effective speech data, and filters out the junk sound segments, thereby obtaining effective speech data with better quality.

[0059] in, Figure 3 FIG. 1 is a flow chart of the junk sound filtering method provided by the present invention. Figure 3 As shown, the junk sound filtering method includes the following steps: Step 310: After valid voice data is determined from the audio data to be recognized, voiceprint features of each segment in the valid voice data are extracted; Step 320: based on the similarity between the voiceprint features of each segment and the noise garbage voiceprint features, and the similarity between the voiceprint features of each segment and the distortion signal garbage voiceprint features, filter out the garbage sound segments from the valid voice data; The noise garbage sound voiceprint feature is determined based on the voiceprint features corresponding to multiple sample noise garbage sounds, and the distorted signal garbage sound voiceprint feature is determined based on the voiceprint features corresponding to multiple sample distorted signal garbage sounds.

[0060] Specifically, the voiceprint features of each segment in the valid voice data refer to the voice features corresponding to the sound in each segment, and the voiceprint features may include features such as sound quality, sound length, and pitch. Based on the features such as sound quality, sound length, and pitch, the quality of the corresponding valid voice can be determined, and then it can be determined whether the valid voice is distorted voice.

[0061] The voiceprint features of noise garbage sound refer to the speech features corresponding to the sound generated by noise. The voiceprint features corresponding to the sound generated by noise are usually manifested as unstable frequency, amplitude changes, irregular harmonic structure, etc.

[0062] In order to determine the voiceprint features of noise and garbage sounds, multiple sample noise and garbage sounds may be collected and their voiceprint features may be extracted. These voiceprint features may include sound quality, sound length, pitch, spectrum characteristics, etc. The voiceprint features of noise and garbage sounds may be determined based on the following formula: in, Indicates the noise and garbage sound voiceprint characteristics, Indicates The voiceprint features of the noise and garbage sounds of samples, Indicates the total amount of noise garbage in the sample.

[0063] In addition, the voiceprint features of distorted signal garbage sound refer to the speech features corresponding to the sound generated by signal distortion. The voiceprint features corresponding to the distorted sound are usually manifested as sound quality cracking, pitch change, spectrum distortion, etc. In order to determine the voiceprint features of distorted signal garbage sound, multiple sample distorted signal garbage sounds can also be collected and their voiceprint features can be extracted. These features can also include sound quality, sound length, pitch, spectrum characteristics, etc. Among them, the voiceprint features of distorted signal garbage sound can be determined based on the following formula: in, Indicates the voiceprint characteristics of distorted signals and junk sounds. Indicates The voiceprint features of the distorted signal garbage sound of samples, Indicates the total amount of garbage sound in the sample distortion signal.

[0064] Furthermore, the similarity between the voiceprint features of each segment and the voiceprint features of noise and garbage sound is used to characterize the matching degree between the voiceprint features of each segment and the voiceprint features of noise and garbage sound. The higher the similarity, the higher the matching degree between the voiceprint features of the corresponding segment and the voiceprint features of noise and garbage sound, and the higher the probability that the corresponding segment is noise and garbage sound.

[0065] Similarly, the similarity between the voiceprint features of each segment and the voiceprint features of the distorted signal garbage sound is used to characterize the matching degree between the voiceprint features of each segment and the voiceprint features of the distorted signal garbage sound. The higher the similarity, the higher the matching degree between the voiceprint features of the corresponding segment and the voiceprint features of the distorted signal garbage sound, and the higher the probability that the corresponding segment is the distorted signal garbage sound. The above similarity can be measured by cosine similarity or Euclidean distance, which is not specifically limited in the embodiment of the present invention.

[0066] Based on this, segments with high similarity to noise garbage voiceprint features and / or segments with high similarity to distorted signal garbage voiceprint features can be filtered out as garbage segments, thereby further ensuring the quality of valid voice data.

[0067] In addition, the junk audio segments are filtered out here by using voiceprint features, which does not require a complicated training process and saves training costs.

[0068] Based on any of the above embodiments, based on the similarity between the voiceprint features of each segment and the noise garbage voiceprint features, and the similarity between the voiceprint features of each segment and the distorted signal garbage voiceprint features, filtering out garbage sound segments from valid voice data includes: If the similarity between the voiceprint features of any segment and the noise garbage voiceprint features is greater than or equal to a threshold, and / or the similarity between the voiceprint features of any segment and the distorted signal garbage voiceprint features is greater than or equal to a threshold, then any segment is treated as a garbage segment and the garbage segment is filtered out from the valid speech data.

[0069] Specifically, if the similarity between the voiceprint feature of any segment and the noise garbage voiceprint feature is greater than or equal to a threshold, it indicates that the probability that the corresponding segment is a noise garbage segment is high, that is, the segment can be regarded as a noise garbage segment.

[0070] If the similarity between the voiceprint feature of any segment and the voiceprint feature of the distorted signal garbage sound is greater than or equal to the threshold, it indicates that the probability that the corresponding segment is a distorted signal garbage sound segment is high, that is, the segment can be regarded as a distorted signal garbage sound segment.

[0071] If the similarity between the voiceprint features of any segment and the noise garbage sound voiceprint features is greater than or equal to the threshold, and the similarity between the voiceprint features of any segment and the distorted signal garbage sound voiceprint features is greater than or equal to the threshold, it indicates that the corresponding segment contains both noise garbage sound and distorted signal garbage sound, that is, the corresponding segment can also be regarded as a garbage sound segment.

[0072] After determining that the corresponding segment is a junk sound segment, the junk sound segment is filtered out from the valid voice data, thereby obtaining high-quality valid voice data.

[0073] Based on any of the above embodiments, any segment is regarded as a junk sound segment, and then the method further includes: The noise and garbage voiceprint features and / or the distortion and garbage voiceprint features are updated based on the voiceprint features of any segment.

[0074] Specifically, if any segment is determined to be a garbage sound segment, the noise garbage sound voiceprint features and / or the distorted signal garbage sound voiceprint features can be updated based on the segment to dynamically update the noise garbage sound voiceprint features and the distorted signal garbage sound voiceprint features, thereby ensuring the reliability of the garbage sound voiceprint features and the distorted signal garbage sound voiceprint features.

[0075] Exemplarily, if any segment is determined to be a noise junk sound segment, the noise junk sound voiceprint feature is updated based on the voiceprint feature of the segment, such as the segment can be used as a sample noise junk sound, and the sample noise junk sound is used to update the above formula for solving the noise junk sound voiceprint feature, and the updated noise junk sound voiceprint feature is obtained. If any segment is determined to be a distorted signal junk sound, the distorted signal junk sound voiceprint feature is updated based on the voiceprint feature of the segment, such as the segment can be used as a sample distorted signal junk sound, and the sample distorted signal junk sound is used to update the above formula for solving the distorted signal junk sound voiceprint feature, and the updated distorted signal junk sound voiceprint feature is obtained.

[0076] Based on any of the above embodiments, the audio data to be recognized is collected in a variety of ways; Determine valid voice data from the audio data to be recognized, and then include: Based on the duration of the valid voice data corresponding to the audio data to be recognized under each acquisition mode, the target voice data is determined from each valid voice data.

[0077] Specifically, the audio data to be recognized can be collected in a variety of ways, for example, the audio data to be recognized can be collected by microphone recording, telephone recording, network audio recording, mobile phone recording, etc. Considering that different collection methods may bring different degrees of noise, distortion and other problems, the embodiment of the present invention uses the above-mentioned embodiment method to perform effective speech recognition on the audio data to be recognized under each collection method, and then counts the duration of the audio data to be recognized corresponding to the effective speech data under each collection method. The longer the duration, the higher the quality of the effective speech data under the corresponding collection method. In order to filter out the optimal speech data from multiple effective speech data, the effective speech data under the collection method with the longest duration can be used as the target speech data, so that based on the high-quality target speech data, tasks such as speech recognition and semantic understanding can be effectively performed.

[0078] Based on any of the above embodiments, the present invention further provides an effective speech recognition method, the method comprising: Determine audio data to be recognized, extract audio features of the audio data to be recognized based on an effective speech recognition model, and apply the audio features of the audio data to be recognized to determine effective speech data from the audio data to be recognized.

[0079] The effective speech recognition model is trained based on training samples and corresponding effective speech tags. The training samples include sample audio data, noisy sample audio data, and pure noise data. The effective speech duration in the training samples is not less than 500 hours. The noisy sample audio data can be a sample audio data with a random increase of -5 to 15dB of noise.

[0080] After obtaining the training samples, the training samples can be segmented, such as randomly segmented into segments of 5s to 20s. Then, the segmented training samples are input into the pre-training network in batches (the pre-training network can be a CPC neural network, a wav2vec2.0 neural network, etc.), and the pre-training network extracts the audio features of the segmented training samples, and the audio features of the segmented training samples are input into the DNN network, and the DNN network performs effective speech prediction based on the audio features to obtain effective predicted speech. Among them, the pre-training network and the DNN network constitute the initial model.

[0081] Based on the difference between the effective predicted speech and the effective speech label, the classification loss value is determined; based on the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, the intra-class loss value is determined; based on the distance between the audio features of the sample audio data and the audio features of the pure noise data, the inter-class loss value is determined; based on the preset weights, the classification loss value, the intra-class loss value, and the inter-class loss value are weighted and added to obtain the training loss value. Based on the training loss value, the parameters of the initial model are updated to obtain an effective speech recognition model.

[0082] In addition, after determining the valid speech data from the audio data to be identified, the voiceprint features of each segment in the valid speech data are extracted; if the similarity between the voiceprint features of any segment and the noise garbage sound voiceprint features is greater than or equal to a threshold, and / or the similarity between the voiceprint features of any segment and the distorted signal garbage sound voiceprint features is greater than or equal to a threshold, then any segment is treated as a garbage sound segment and the garbage sound segment is filtered out from the valid speech data.

[0083] The noise garbage voiceprint feature is determined based on the voiceprint features corresponding to multiple sample noise garbage sounds, and the distorted signal garbage voiceprint feature is determined based on the voiceprint features corresponding to multiple sample distorted signal garbage sounds. In addition, if any segment is regarded as a garbage segment, the noise garbage voiceprint feature and / or the distorted signal garbage voiceprint feature are updated based on the voiceprint features of any segment.

[0084] Finally, based on the duration of the valid voice data corresponding to the audio data to be recognized under each acquisition mode, the optimal target voice data is screened from each valid voice data.

[0085] The effective speech recognition device provided by the present invention is described below. The effective speech recognition device described below and the effective speech recognition method described above can be referenced to each other.

[0086] Based on any of the above embodiments, Figure 4 Schematic diagram of the structure of the effective speech recognition device provided by the present invention. Figure 4 As shown, the device comprises: A determination unit 410, configured to determine audio data to be identified; The recognition unit 420 is used to extract the audio features of the audio data to be recognized based on the effective speech recognition model, and apply the audio features of the audio data to be recognized to determine the effective speech data from the audio data to be recognized; The effective speech recognition model is trained with the objectives of minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data. The effective predicted speech is obtained by the effective speech recognition model performing effective speech recognition on the sample audio data.

[0087] Based on any of the above embodiments, the training steps of the effective speech recognition model include: Based on the initial model, extract the audio features of the sample audio data, the audio features of the sample audio data after adding noise, and the audio features of the pure noise data; Based on the initial model, applying the audio features of the sample audio data to determine the effective predicted speech; Determine a training loss value based on the difference between the valid predicted speech and the valid speech label, the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and the distance between the audio features of the sample audio data and the audio features of the pure noise data; Based on the training loss value, the parameters of the initial model are updated to obtain an effective speech recognition model.

[0088] Based on any of the above embodiments, determining a training loss value based on a difference between the effective predicted speech and the effective speech label, a distance between an audio feature of the sample audio data and an audio feature of the noisy sample audio data, and a distance between an audio feature of the sample audio data and an audio feature of pure noise data, comprises: Determining a classification loss value based on the difference between the valid predicted speech and the valid speech label; Determining an intra-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of the noisy sample audio data; Determining an inter-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of pure noise data; Based on the preset weights, the classification loss value, intra-class loss value, and inter-class loss value are weighted added to obtain the training loss value.

[0089] Based on any of the above embodiments, valid voice data is determined from the audio data to be recognized, and then the method further includes: Extract the voiceprint features of each segment in the valid voice data; Based on the similarity between the voiceprint features of each segment and the voiceprint features of noise garbage sound, and the similarity between the voiceprint features of each segment and the voiceprint features of distorted signal garbage sound, the garbage sound segments are filtered out from the valid voice data; The noise garbage sound voiceprint feature is determined based on the voiceprint features corresponding to multiple sample noise garbage sounds, and the distorted signal garbage sound voiceprint feature is determined based on the voiceprint features corresponding to multiple sample distorted signal garbage sounds.

[0090] Based on any of the above embodiments, based on the similarity between the voiceprint features of each segment and the noise garbage voiceprint features, and the similarity between the voiceprint features of each segment and the distorted signal garbage voiceprint features, filtering out garbage sound segments from valid voice data includes: If the similarity between the voiceprint features of any segment and the noise garbage voiceprint features is greater than or equal to a threshold, and / or the similarity between the voiceprint features of any segment and the distorted signal garbage voiceprint features is greater than or equal to a threshold, then any segment is treated as a garbage segment and the garbage segment is filtered out from the valid speech data.

[0091] Based on any of the above embodiments, any segment is regarded as a junk sound segment, and then the method further includes: The noise and garbage voiceprint features and / or the distortion and garbage voiceprint features are updated based on the voiceprint features of any segment.

[0092] Based on any of the above embodiments, the audio data to be recognized is collected in a variety of ways; Determine valid voice data from the audio data to be recognized, and then include: Based on the duration of the valid voice data corresponding to the audio data to be recognized under each acquisition mode, the target voice data is determined from each valid voice data.

[0093] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the effective speech recognition method, which includes: determining the audio data to be recognized; extracting the audio features of the audio data to be recognized based on the effective speech recognition model, and applying the audio features of the audio data to be recognized to determine the effective speech data from the audio data to be recognized; the effective speech recognition model takes minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the sample audio data after adding noise, and maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data as the training objectives, and the effective predicted speech is obtained by the effective speech recognition model performing effective speech recognition on the sample audio data.

[0094] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0095] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the effective speech recognition method provided by the above-mentioned methods, which includes: determining audio data to be recognized; based on an effective speech recognition model, extracting audio features of the audio data to be recognized, and applying the audio features of the audio data to be recognized to determine effective speech data from the audio data to be recognized; the effective speech recognition model takes minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the sample audio data after noise addition, and maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data as training objectives, and the effective predicted speech is obtained by the effective speech recognition model performing effective speech recognition on the sample audio data.

[0096] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the effective speech recognition method provided by the above-mentioned methods, the method comprising: determining audio data to be recognized; extracting audio features of the audio data to be recognized based on an effective speech recognition model, and applying the audio features of the audio data to be recognized to determine effective speech data from the audio data to be recognized; the effective speech recognition model takes minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the sample audio data after noise addition, and maximizing the distance between the audio features of the sample audio data and the audio features of the pure noise data as training objectives, and the effective predicted speech is obtained by the effective speech recognition model performing effective speech recognition on the sample audio data.

[0097] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An effective speech recognition method, characterized in that: include: Determining audio data to be recognized; Extracting audio features of the audio data to be recognized based on the effective speech recognition model, and applying the audio features of the audio data to be recognized to determine effective speech data from the audio data to be recognized; The effective speech recognition model has the training objectives of minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and maximizing the distance between the audio features of the sample audio data and the audio features of pure noise data. The effective predicted speech is obtained by the effective speech recognition model through effective speech recognition of the sample audio data.

2. The effective speech recognition method according to claim 1, characterized in that: The training steps of the effective speech recognition model include: Based on the initial model, extracting audio features of the sample audio data, audio features of the noisy sample audio data, and audio features of the pure noise data; Based on the initial model, applying the audio features of the sample audio data, determining the effective predicted speech; Determine a training loss value based on the difference between the effective predicted speech and the effective speech label, the distance between the audio feature of the sample audio data and the audio feature of the noisy sample audio data, and the distance between the audio feature of the sample audio data and the audio feature of the pure noise data; Based on the training loss value, the parameters of the initial model are updated to obtain the effective speech recognition model.

3. The effective speech recognition method according to claim 2, characterized in that: The determining of the training loss value based on the difference between the effective predicted speech and the effective speech label, the distance between the audio feature of the sample audio data and the audio feature of the noisy sample audio data, and the distance between the audio feature of the sample audio data and the audio feature of the pure noise data comprises: Determining a classification loss value based on a difference between the valid predicted speech and the valid speech label; Determining an intra-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of the noisy sample audio data; Determining an inter-class loss value based on a distance between an audio feature of the sample audio data and an audio feature of pure noise data; Based on preset weights, the classification loss value, the intra-class loss value, and the inter-class loss value are weightedly added to obtain the training loss value.

4. The effective speech recognition method according to any one of claims 1 to 3, characterized in that: The determining of valid voice data from the audio data to be recognized further comprises: Extracting voiceprint features of each segment in the valid voice data; Based on the similarity between the voiceprint features of each segment and the noise garbage voiceprint features, and the similarity between the voiceprint features of each segment and the distorted signal garbage voiceprint features, filtering out the garbage sound segments from the valid voice data; The noise and garbage sound voiceprint features are determined based on the voiceprint features corresponding to a plurality of sample noise and garbage sounds, and the distortion signal and garbage sound voiceprint features are determined based on the voiceprint features corresponding to a plurality of sample distortion signal and garbage sounds.

5. The effective speech recognition method according to claim 4, characterized in that: The filtering out junk sound segments from the valid voice data based on the similarity between the voiceprint features of each segment and the noise junk sound voiceprint features, and the similarity between the voiceprint features of each segment and the distorted signal junk sound voiceprint features, comprises: If the similarity between the voiceprint features of any segment and the noise garbage voiceprint features is greater than or equal to a threshold, and / or the similarity between the voiceprint features of any segment and the distorted signal garbage voiceprint features is greater than or equal to the threshold, then any segment is taken as the garbage segment and the garbage segment is filtered out from the valid voice data.

6. The effective speech recognition method according to claim 5, characterized in that: The method of using any of the segments as the junk sound segment further comprises: The noise and garbage voiceprint features are updated based on the voiceprint features of any one of the segments and / or the distorted signal and garbage voiceprint features are updated.

7. The effective speech recognition method according to any one of claims 1 to 3, characterized in that: The audio data to be identified is collected in a variety of ways; The determining of valid voice data from the audio data to be recognized further comprises: Based on the duration of the valid voice data corresponding to the audio data to be recognized under each acquisition mode, the target voice data is determined from each valid voice data.

8. An effective speech recognition device, characterized in that: include: A determination unit, used to determine the audio data to be identified; A recognition unit, configured to extract audio features of the audio data to be recognized based on a valid speech recognition model, and determine valid speech data from the audio data to be recognized by applying the audio features of the audio data to be recognized; The effective speech recognition model has the training objectives of minimizing the difference between the effective predicted speech and the effective speech label, minimizing the distance between the audio features of the sample audio data and the audio features of the noisy sample audio data, and maximizing the distance between the audio features of the sample audio data and the audio features of pure noise data. The effective predicted speech is obtained by the effective speech recognition model through effective speech recognition of the sample audio data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the effective speech recognition method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the effective speech recognition method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the effective speech recognition method according to any one of claims 1 to 7 is implemented.

Images