An audio processing method, apparatus, storage medium, and electronic device

Abstract

Embodiments of the present disclosure disclose an audio processing method, device, storage medium and electronic device. The audio processing method comprises: obtaining an audio frame to be processed, determining an audio type of the audio frame based on a current recognition threshold; in the case that a current audio frame meets a threshold adjustment condition, determining a determination state of the recognized audio type based on feature information of the recognized continuous audio frame; and adjusting the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for recognizing the audio type of the next audio frame. Through the above technical solution, the dynamic adjustment of the recognition threshold is realized, and then the audio type is recognized according to the dynamically adjusted recognition threshold, which can improve the accuracy of audio type recognition.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of data processing, and particularly relate to an audio processing method and device, a storage medium and an electronic device. BACKGROUND

[0002] With the continuous development of the Internet and communication technology, in the field of communication systems, speech recognition, etc., audio recognition is increasingly concerned by users.

[0003] At present, the method of setting a fixed threshold can be used for audio recognition, but the recognition accuracy of this method is poor. SUMMARY

[0004] Embodiments of the present disclosure provide an audio processing method and device, a storage medium and an electronic device to improve the accuracy of audio recognition.

[0005] In a first aspect, the embodiments of the present disclosure provide an audio processing method, comprising:

[0006] obtaining an audio frame to be processed, and determining an audio type of the audio frame based on a current recognition threshold;

[0007] in a case where a current audio frame meets a threshold adjustment condition, determining a determination state of a recognized audio type based on feature information of a continuous audio frame that has been recognized;

[0008] adjusting the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for recognizing the audio type of a next audio frame.

[0009] In a second aspect, the embodiments of the present disclosure further provide an audio processing device, comprising:

[0010] a type determination module configured to obtain an audio frame to be processed, and determine an audio type of the audio frame based on a current recognition threshold;

[0011] a state determination module configured to, in a case where a current audio frame meets a threshold adjustment condition, determine a determination state of a recognized audio type based on feature information of a continuous audio frame that has been recognized;

[0012] a threshold adjustment module configured to adjust the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for recognizing the audio type of a next audio frame.

[0013] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising:

[0014] one or more processors;

[0015] a storage device configured to store one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method according to any of the embodiments of the present disclosure.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions for executing the audio processing method according to any of the embodiments of the present disclosure when executed by a computer processor.

[0018] The technical scheme of the embodiments of the present disclosure is to obtain an audio frame to be processed, which indicates that the scheme is to process audio data in frames, so as to determine the audio type of the audio frame based on a current recognition threshold, to realize preliminary judgment of the audio frame type; in the case that the current audio frame meets the threshold adjustment condition, the determination state of the recognized audio type is determined based on the feature information of the recognized continuous audio frame, which can realize re-verification of the audio type of the recognized continuous audio frame, and the current recognition threshold is adjusted according to the determination state of the re-verification, and the adjusted recognition threshold can be used for recognition of the audio type of the next audio frame, that is, the recognition threshold is continuously adjusted in the process of recognizing the next audio frame, so as to realize dynamic adjustment of the recognition threshold, and then the audio category is recognized according to the dynamically adjusted recognition threshold, which can improve the accuracy of audio type recognition. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0020] Figure 1 is a flowchart of an audio processing method provided by the embodiments of the present disclosure;

[0021] Figure 2 is a flowchart of an audio processing method provided by the embodiments of the present disclosure;

[0022] Figure 3 is a flowchart of an audio processing method provided by the embodiments of the present disclosure;

[0023] Figure 4 is a flowchart of an audio processing method provided by the embodiments of the present disclosure;

[0024] Figure 5 is a structural schematic diagram of an audio processing device provided by the embodiments of the present disclosure;

[0025] Figure 6 is a structural schematic diagram of an electronic device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted in a limited sense as set forth in the embodiments set forth herein, but rather, the embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that each step described in the method embodiments of the present disclosure can be performed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions will be given in the description below.

[0029] It should be noted that the "first", "second", and the like concepts mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modification of "one" or "multiple" mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0031] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure is shown. The embodiment of the present disclosure is suitable for audio type recognition according to an automatically adjusted threshold. The method can be performed by an audio processing device provided by the embodiment of the present disclosure. The audio processing device can be implemented in the form of software and / or hardware, and can be implemented by an electronic device, which can be a mobile terminal or a PC, etc.

[0032] As Figure 1 , the method of the present embodiment comprises:

[0033] S110, obtaining an audio frame to be processed, and determining an audio type of the audio frame based on a current recognition threshold.

[0034] S120, in a case where the current audio frame meets a threshold adjustment condition, determining a determination state of the identified audio type based on the feature information of the identified continuous audio frame.

[0035] S130, adjusting the current identification threshold according to the determination state, wherein the adjusted identification threshold is used for audio type identification of a next audio frame.

[0036] In the embodiments of the present disclosure, the execution subject (the electronic device described above) for executing the audio processing method includes but is not limited to a mobile phone, a smart watch, a computer and the like. The electronic device described above can obtain the audio frame to be processed in various ways. For example, audio data can be collected in real time through an audio collection device, and the audio frame to be processed can be extracted from the audio data. Alternatively, the audio data can be called from a preset storage location or other devices, and the audio frame to be processed can be extracted from the audio data. The embodiments of the present disclosure do not limit the method for obtaining the audio frame to be processed. The audio data described above can include but is not limited to call audio data, audio data in a video, live audio data and the like, which are not limited.

[0037] The audio data can be a piece of audio, which can contain voice, noise and the like. In the embodiments, the duration of the audio data is not limited. In order to improve the identification accuracy of the audio data and improve the real-time processing of the audio data, the audio data is divided into a plurality of audio frames, and each audio frame is identified and processed. In a case where the audio data is real-time data, the audio frames are sequentially divided from the real-time collected audio data, and the obtained audio frames are identified and processed in real time. In a case where the audio data is offline data, each audio frame can be processed according to the time sequence of the divided audio frames. The audio frame can be audio data with a preset time length, wherein the duration of the audio frame can be determined according to the identification accuracy, which is not limited. For example, the duration of the audio frame can be 10 ms. In the embodiments, the audio frame to be processed can be a current audio frame to be processed.

[0038] In the embodiments, the audio type of each voice frame is identified based on a dynamic identification threshold. For the audio frame to be processed, a current identification threshold corresponding to the processing time of the audio frame to be processed is obtained. The current identification threshold can be a judgment threshold corresponding to the processing time of the current audio frame, which can be used to determine the audio type of the audio frame. It should be noted that the identification threshold is an adjustable value. The current identification threshold can be an identification threshold adjusted on the basis of the previous voice frame, or can also be an initial identification threshold without adjustment.

[0039] In this embodiment, the audio frame is identified by audio type, which can be set according to the identification requirements. In some embodiments, the audio type is divided into speech type and noise type according to whether the audio frame contains noise. In some embodiments, the audio frame can be divided into human voice, rain sound, bird sound, etc. according to the object that emits the sound. In some embodiments, the audio frame can be divided into songs, speeches, etc. without limitation. Optionally, the audio type includes speech type and noise type. The audio frame corresponding to the speech type is a speech frame, which can include speech information. The speech information can be real language content. The audio frame corresponding to the noise type is a noise frame, which can include noise information. The noise information can be irrelevant interference information such as interference sound in the environment.

[0040] On the basis of the above-mentioned embodiments, the audio frame can be identified based on a pre-set audio identification algorithm to obtain an identification probability of the audio frame, wherein the pre-set audio identification algorithm is an identification algorithm suitable for the identification requirements of the audio type. In some embodiments, the audio identification algorithm can be a machine learning model, which can be a neural network model, etc. without limitation. The identification probability of the audio frame is judged based on the current identification threshold to determine the audio type of the audio frame. Taking the case where the audio type includes speech type and noise type as an example, in the case where the identification probability of the audio frame is greater than or equal to the current identification threshold, the audio frame is determined to be a speech frame, and in the case where the identification probability of the audio frame is less than the current identification threshold, the audio frame is determined to be a noise frame. It should be noted that the identification threshold can be one or more data, and the number of identification thresholds and the judgment method of the identification probability of the audio frame can be set according to the judgment requirements of the audio type without limitation.

[0041] In this embodiment, the identification threshold can be dynamically adjusted, which can be adjusted in real time by the determination state of the identified speech frame. The adjusted identification threshold is more suitable for the judgment state of the next audio frame, so that the accuracy of the audio type identified by the adjusted identification threshold is higher. Further, the adjusted identification threshold can be used to identify the audio type of the next audio frame, that is, the identification threshold is continuously adjusted in the process of identifying the next audio frame, so as to realize the dynamic adjustment of the identification threshold to adapt to different environments and improve the accuracy of the audio type identification.

[0042] It should be noted that after the identification of each voice frame is completed, it is determined whether the threshold adjustment condition is met, in the case where the threshold adjustment condition is met, the current identification threshold is adjusted, the next audio frame is identified based on the adjusted identification threshold, in the case where the threshold adjustment condition is not met, the current identification threshold is kept unchanged, the next audio frame is identified based on the current identification threshold, avoiding frequent adjustment of the identification threshold and interference of the frequent adjustment of the identification threshold on the audio type identification.

[0043] In the embodiments of the present disclosure, the threshold adjustment condition can be a condition set by using the characteristics of the continuous audio frames of the voice type, and can include but is not limited to a condition for judging whether the voice is ended, that is, the threshold adjustment condition can be used to screen out the voice segment that has been interrupted or paused. Optionally, the threshold adjustment condition is that the audio type of the current audio frame is a noise type, and the audio type of the previous audio frame is a voice type.

[0044] It can be understood that in a segment of audio, if the audio type of the current audio frame is a noise type and the audio type of the previous audio frame is a voice type, it indicates that there can be a voice interruption in the segment of audio, that is, a speech pause or termination and the like. The threshold adjustment condition is used to judge whether a segment of voice type audio is ended, in the case where a segment of voice type audio is ended, the segment of voice type audio is verified again, that is, the overall verification of multiple audio frames is performed, whether the audio is a voice is considered from the overall audio, to obtain a judgment state of the identified audio type, to measure whether the current identification threshold is set appropriately. The judgment method is more reliable, and can avoid the occurrence of a single audio frame misjudging the audio type.

[0045] Among them, the identified continuous audio frame can be a segment of continuous audio that has completed audio type identification, that is, multiple audio frames that are continuously identified as a voice type, the identified continuous audio frame can be an audio segment closest to the end of a voice located before the current audio frame, that is, the last audio frame of the identified continuous audio frame is the previous audio frame of the current audio frame. The feature information of the identified continuous audio frame refers to reference information for judging whether the identified audio type is correct, and the feature information can be obtained by calculating or statistically processing the basic information of the identified continuous audio frame. The identified audio type refers to the overall type of the continuous audio frame, that is, the audio type corresponding to the identified segment of audio, rather than the audio type of a single audio frame. The judgment state refers to the right or wrong state of the judgment of the identified audio type.

[0046] On the basis of the above-mentioned embodiments, the feature information includes one or more of the following: length of continuous voice frame, identification probability, fundamental frequency and energy value.

[0047] The length of the continuous speech frame refers to the time length of the continuous speech frame, which can be obtained by counting, i.e., the time length of each audio frame in the continuous speech frame. The recognition probability refers to the probability that the continuous speech frame is of the speech type, which can specifically be the average probability that each audio frame in the continuous speech frame is of the speech type. The pitch frequency refers to the frequency of vocal cord vibration, and specifically, the pitch frequency of the continuous speech frame can be the average pitch frequency of each audio frame in the continuous speech frame. The energy value of the continuous speech frame can be the sum or average of the energy values of each audio frame in the continuous speech frame, etc. It should be noted that the above characteristic information can be obtained by counting, calculating, etc., and will not be described here.

[0048] For example, the valid speech segment (i.e., the continuous audio frame of the speech type) has the characteristics of a long time length, a large recognition probability, a high pitch frequency, or a large energy value. The characteristic information of the continuous audio frame can be judged by setting a time length discrimination threshold, a probability discrimination threshold, a frequency discrimination threshold, or an energy discrimination threshold to determine whether the determination result of the audio type of the continuous audio frame is correct to obtain the determination state of the recognized audio type. Specifically, if the characteristic information is greater than the corresponding set threshold, it indicates that the determination state is correct, otherwise, it indicates that the determination state is incorrect.

[0049] In some optional implementations of the embodiments of the present disclosure, determining the determination state of the recognized audio type based on the characteristic information of the recognized continuous audio frame comprises: determining the characteristic information of the continuous speech frame before the current audio frame, comparing the characteristic information with a judgment threshold of the characteristic information; and determining the determination state of the recognized audio type based on the comparison result.

[0050] Specifically, the characteristic information of the continuous speech frame before the current audio frame can be obtained by calculation, statistics, etc., and can include one or more, i.e., the continuous speech frame before the current audio frame can have one or more characteristic information. It can be understood that when the characteristic information is multiple, the evaluation parameters of the determination state are more abundant, which can improve the accuracy of the determination state. The judgment threshold of the characteristic information can be set according to experience, and can include one or more, i.e., the characteristic information can have one or more judgment thresholds. After the characteristic information and the corresponding judgment threshold are determined, the characteristic information is compared with the judgment threshold of the characteristic information one by one to obtain a comparison result, and then the determination state of the recognized audio type is determined according to the comparison result, wherein the comparison result and the determination state have a mapping relationship. Optionally, if each characteristic information is greater than the judgment threshold of the characteristic information, the determination state of the recognized audio type is correct; otherwise, the determination state of the recognized audio type is incorrect. In this embodiment, the state is determined by threshold comparison, which is simple and efficient, and can quickly verify the audio type of the recognized continuous speech frame.

[0051] In some optional embodiments, the length of the continuous speech frame and the average recognition probability are determined, the length of the continuous speech frame is judged based on a time length threshold, the average recognition probability is judged based on a recognition probability threshold, and the determination state of the recognized audio type is determined as correct in the case that the length of the continuous speech frame is greater than the time length threshold and the average recognition probability is greater than the recognition probability threshold, and the determination state of the recognized audio type is determined as incorrect in the case that the length of the continuous speech frame is less than or equal to the time length threshold and / or the average recognition probability is less than or equal to the recognition probability threshold.

[0052] In some optional implementations of the embodiments of the present disclosure, the determination state includes an incorrect state and a correct state, and the current recognition threshold is adjusted according to the determination state, where the adjustment manner of the current recognition threshold includes increasing and decreasing, specifically, the current recognition threshold is increased in the case that the determination state is the incorrect state, and the current recognition threshold is decreased in the case that the determination state is the correct state.

[0053] The incorrect state indicates that the determination of the recognized audio type is incorrect, indicating that noise with a short time length, a small recognition probability, a low pitch frequency or a small energy value is identified as speech, and thus the current recognition threshold needs to be increased to prevent the noise from being identified as speech in the future. The correct state indicates that the determination of the recognized audio type is correct, and the current recognition threshold can be decreased in the case that the determination state is the correct state, so as to relax the discrimination standard of the recognition probability. It should be noted that if the current recognition threshold is increased to an upper threshold or decreased to a lower threshold, the current recognition threshold will not continue to be increased or decreased, so as to prevent over-adjustment and cause the audio recognition accuracy to be correct.

[0054] The technical scheme of the embodiments of the present disclosure includes obtaining an audio frame to be processed, indicating that the scheme is used for processing audio data in frames, so as to determine the audio type of the audio frame based on the current recognition threshold, and to realize preliminary judgment of the audio frame type. In the case that the current audio frame meets the threshold adjustment condition, the determination state of the recognized audio type is determined based on the feature information of the recognized continuous audio frame, so as to realize re-verification of the audio type of the recognized continuous audio frame. The current recognition threshold is adjusted according to the determination state of the re-verification, and the adjusted recognition threshold can be used for recognizing the audio type of the next audio frame, that is, the recognition threshold is continuously adjusted in the process of recognizing the next audio frame, so as to realize dynamic adjustment of the recognition threshold, and then the audio category is recognized according to the dynamically adjusted recognition threshold, so as to improve the accuracy of the audio type recognition.

[0055] Reference Figure 2 , Figure 2A flowchart of an audio processing method provided by an embodiment of the present disclosure is shown in FIG. 2. The method of the present embodiment can be combined with the various optional schemes of the audio processing method provided in the above embodiments. The audio processing method provided by the present embodiment is further refined. Optionally, determining the audio type of the audio frame based on the current recognition threshold comprises: extracting an audio feature of the audio frame, inputting the audio feature into an audio recognition model to obtain a recognition probability of the audio frame; and determining the audio type of the audio frame based on the current recognition threshold and the recognition probability.

[0056] As Figure 2 the method of the present embodiment comprises:

[0057] S210, obtaining an audio frame to be processed.

[0058] S220, extracting an audio feature of the audio frame, inputting the audio feature into an audio recognition model to obtain a recognition probability of the audio frame.

[0059] S230, determining the audio type of the audio frame based on the current recognition threshold and the recognition probability.

[0060] S240, in the case where the current audio frame meets the threshold adjustment condition, determining a determination state of the recognized audio type based on feature information of the recognized continuous audio frames.

[0061] S250, adjusting the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for recognizing the audio type of the next audio frame.

[0062] The audio feature can be used to determine the recognition probability of the audio frame, and the audio feature can be obtained through a series of calculations and transformations. The audio feature extraction process can include but is not limited to windowing, fast Fourier transform, mel spectrum conversion, and normalization. The audio recognition model can be a trained deep learning model, for example, including but not limited to a recurrent neural network model, a long short-term memory recurrent neural network, etc. The recognition probability of the audio frame can be used to determine whether the audio frame is speech.

[0063] For example, in a specific application scenario, the length of an audio frame can be 10 ms, the sampling rate can be 16 KHz, and an audio frame input is x(n), where n = 0, 1, …, 160, that is, 160 sample point data. When windowing, the signal x(n) is multiplied by a window function w(n) to obtain the windowed signal x(n)*w(n). The window function can be a Hanning window function. When performing fast Fourier transform, the windowed signal is subjected to fast Fourier transform to obtain the signal spectrum X(m) = FFT(x(n)*w(n)). When calculating the mel spectrum, the spectrum is divided into several segments, the energy of each segment is summed based on the mel filter function to obtain the mel spectrum, and the mel spectrum is normalized to obtain the audio feature of the audio frame. The audio feature is input into the audio recognition model to obtain the recognition probability of the audio frame.

[0064] Further, if the recognition probability is greater than or equal to the current recognition threshold, the audio type of the audio frame is a speech type, and if the recognition probability is less than the current recognition threshold, the audio type of the audio frame is a noise type.

[0065] On the basis of the above embodiment, the training method of the audio recognition model comprises: obtaining a noise-free audio, setting a label for each audio segment in the noise-free audio; obtaining noise information, superimposing the noise information into the noise-free audio to form a sample audio, wherein the noise information comprises at least one of steady-state noise, transient noise and howling noise; and iteratively training the audio recognition model to be trained based on the sample audio until a trained audio recognition model is obtained.

[0066] The howling noise is a kind of feedback sound. For example, in a call scenario, if the sending end and the receiving end devices are in the same physical space, howling phenomenon is easy to occur. Traditional voice activity detection algorithms are difficult to identify howling, and therefore do not have the ability to suppress howling. The embodiment superimposes the howling noise into the noise-free audio, and uses the audio superimposed with the howling noise as a sample audio to train the audio recognition model, so that the audio recognition model has the ability to determine howling as noise, and therefore the audio processing method has the ability to suppress howling. Further, by adding various noise sample audios such as steady-state noise, transient noise and howling noise into the noise-free audio to train the audio recognition model, the audio recognition model has the ability to identify various noises such as steady-state noise, transient noise and howling noise, thereby improving the robustness and applicability of the audio recognition model.

[0067] Exemplarily, the noise-free audio can be clean speech and blank audio, an audio frame greater than a preset threshold in the clean speech can be marked as 1, and other audio frames can be marked as 0. Further, the noise information can be superimposed on the noise-free audio to form a sample audio, and the label of the sample audio is consistent with the label of the noise-free audio, that is, the label does not change.

[0068] In the iterative training process of the audio recognition model, the focal loss can be used as a loss function, and the audio recognition model to be trained is iteratively trained according to the sample audio until a trained audio recognition model is obtained.

[0069] On the basis of the above-mentioned embodiments, the method further comprises adjusting the signal-to-noise ratio in the sample audio; and / or filtering the sample audio based on a preset filter.

[0070] Specifically, by adjusting the signal-to-noise ratio in the sample audio or filtering the sample audio to different degrees according to the preset filter, the number of sample audios can be increased, the diversity of sample audios can be improved, and the trained audio recognition model has stronger generalization ability. The signal-to-noise ratio in the sample audio can be adjusted in a random or fixed manner, and the preset filter can include but is not limited to a high-pass filter and a low-pass filter.

[0071] The technical scheme of the embodiment of the present disclosure adds the howling noise to the noise-free audio, and trains the audio recognition model by using the audio with the added howling noise as a sample audio, so that the audio recognition model has the ability to determine howling as noise, and thus the audio processing method has the ability to suppress howling.

[0072] Reference Figure 3 , Figure 3 The audio processing method flowchart provided by the embodiment of the present disclosure can be combined with each optional scheme in the audio processing method provided in the above-mentioned embodiments. The audio processing method provided by the embodiment is further optimized. Optionally, after obtaining the audio frame to be processed, the audio frame is added to a cache area, wherein the cache area is used to store a plurality of audio frames that have not been output, the current audio frame is located at the last frame of the cache area, and the first frame in the cache area is an output frame. For example, Figure 3 The method of the embodiment comprises:

[0073] S310, obtaining an audio frame to be processed.

[0074] S320, adding the audio frame to a cache area.

[0075] S330, determining the audio type of the audio frame based on a current recognition threshold.

[0076] S340, if the audio type of the current audio frame is the speech type, setting the audio type of each audio frame in the buffer area as the speech type.

[0077] S350, if the audio type of the current audio frame is the noise type, setting the audio type of the last audio frame in the buffer area as the noise type.

[0078] S360, in the case that the current audio frame satisfies the threshold adjustment condition, determining the determination state of the identified audio type based on the feature information of the identified continuous audio frame.

[0079] S370, adjusting the current identification threshold according to the determination state, wherein the adjusted identification threshold is used for identifying the audio type of the next audio frame.

[0080] The buffer area can be used to store a plurality of audio frames that have not been output, and each audio frame is arranged in the buffer area based on a timestamp.

[0081] It should be noted that the embodiment of the present disclosure is a scheme for adding audio frames to the buffer area in real time, that is, while identifying the audio type of the current audio frame, the current audio frame is added to the buffer area. The current audio frame can be stored after the last identified audio frame, that is, the newly added audio frame is always the last frame in the buffer area, and the first frame in the buffer area is set as the output frame, so that the audio can be output in the order of addition, that is, in time sequence, and the situation of incorrect speech playback can be avoided.

[0082] Since the identification threshold is dynamically adjusted during the continuous identification of the audio frame to improve the accuracy of the identification threshold, accordingly, during the identification process of the plurality of audio frames in the initial stage of the audio, the identification threshold is in the initial adjustment stage. In order to avoid the identification error of the plurality of audio frames in the initial stage of the audio, if the audio type of the current audio frame is the speech type, it cannot be determined whether the current audio frame is the initial speech frame, the audio type of each audio frame currently cached in the buffer area is set as the speech type, to avoid the situation that the speech frame is misidentified as a noise frame in the initial identification stage of the audio. If the audio type of the current audio frame is the noise type, the noise frame has no influence on the previous audio frame, and the audio type of the corresponding audio frame of the current audio frame in the buffer area is set as the noise type, that is, the audio type of the last audio frame in the buffer area is set as the noise type. In the embodiment of the present disclosure, by setting the buffer area, a certain delay is set for the audio output, and the audio type of the previous audio frame is updated by the audio type of the subsequent audio frame, to avoid the situation of misidentification of the speech frame during the adjustment of the identification threshold, and to ensure the normal output of the speech frame. The audio in the present embodiment can be real-time output audio such as call audio and live audio.

[0083] In some optional implementations of the embodiments of the present disclosure, after the current recognition threshold is adjusted according to the determination state, the method further includes: determining a current threshold range in which the adjusted recognition threshold is located, and determining the cache area length according to the current threshold range.

[0084] The current threshold range can be set according to experience, and the current threshold range can be compared with the adjusted recognition threshold. Since the recognition threshold is a dynamic value, the comparison result is also a dynamic value, and the cache area length can be dynamically adjusted according to the comparison result. The cache area length refers to the time length of the audio frames that can be cached. It should be noted that the adjustment of the cache area length is actually the adjustment of the audio output delay. In the embodiments of the present disclosure, by dynamically adjusting the cache area length, the situation that the voice at the beginning of the speech is misjudged as noise can be avoided. It can be understood that by adjusting the cache area length, different numbers of audio frames can be cached in the cache, and secondary correction of audio types of different numbers of audio frames can be implemented, thereby improving the accuracy of speech recognition.

[0085] For example, in some embodiments, the current threshold range can include a first threshold and a second threshold, the first threshold is greater than the second threshold, if the adjusted recognition threshold is greater than the first threshold, it indicates that the current recognition error rate of the audio frame is large, and the cache area length is set to a first time length; if the adjusted recognition threshold is less than the second threshold, it indicates that the current recognition error rate of the audio frame is small, and the cache area length is set to a second time length, the first time length can be greater than the second time length; in other cases, the cache area length can remain unchanged. The first time length is greater than the second time length.

[0086] In some embodiments, if the adjusted recognition threshold is less than the first threshold and greater than the second threshold, the cache area length can be set to a third time length, the third time length can be greater than the second time length and less than the first time length. In some embodiments, the current threshold range can include more than two thresholds, and the cache area length is adjusted by the more than two thresholds.

[0087] On the basis of the above-mentioned embodiments, the method further includes: in the case that the audio types of all the audio frames in the cache area are noise types, emptying the cache area, and reconstructing a cache area based on the current cache area length.

[0088] It should be noted that by emptying the cache area, all the noise in the cache area can be cleared, thereby avoiding the situation that the noise is output, and a cache area can be reconstructed according to the current cache area length, so as to continue to cache the subsequent audio frames, thereby avoiding the situation that the voice and the noise are mixed in the cache.

[0089] The technical scheme of the embodiment of the present disclosure can realize the delayed output of the audio frame by adding the audio frame into the buffer area; if the audio type of the current audio frame is the speech type, the audio type of each audio frame in the buffer area is set to the speech type; if the audio type of the current audio frame is the noise type, the audio type of the last audio frame in the buffer area is set to the noise type, so as to realize the judgment of whether the speech ends; further, the length of the buffer area is determined according to the current threshold range, so as to realize the dynamic adjustment of the audio output delay, which can avoid the misjudgment of the beginning of the speech and improve the recognition accuracy of the speech type.

[0090] Reference Figure 4 , Figure 4 The audio processing method flowchart provided by the embodiment of the present disclosure can be combined with the various optional schemes of the audio processing method provided in the above embodiments. The audio processing method provided by the embodiment is further optimized. Optionally, the method further comprises: determining the output gain of the to-be-output audio frame based on the audio type of the to-be-output audio frame; processing the to-be-output audio frame based on the output gain to obtain an output audio frame and outputting the output audio frame. As Figure 4 The method of the embodiment comprises:

[0091] S410, an audio frame to be processed is acquired, and the audio type of the audio frame is determined based on a current recognition threshold.

[0092] S420, in the case where the current audio frame meets a threshold adjustment condition, the determination state of the recognized audio type is determined based on the feature information of the recognized continuous audio frame.

[0093] S430, the current recognition threshold is adjusted according to the determination state, wherein the adjusted recognition threshold is used for the recognition of the audio type of the next audio frame.

[0094] S440, the output gain of the to-be-output audio frame is determined based on the audio type of the to-be-output audio frame.

[0095] S450, the to-be-output audio frame is processed based on the output gain to obtain an output audio frame and output the output audio frame.

[0096] The to-be-output audio frame refers to the audio frame located at the first position of the buffer area. The output gain of the to-be-output audio frame can be used to adjust the output state of the to-be-output audio frame to realize noise reduction.

[0097] Specifically, in some embodiments, a fixed correspondence relationship between the audio type and the output gain can be established in advance, and a fixed relationship table is generated. The audio type of the audio frame to be output is matched in the fixed relationship table to obtain the corresponding audio type. In some embodiments, the output gain of the audio frame to be output can also be determined according to a gain function model. The gain function model can be used to calculate the output gain of the next audio frame to be output using the output gain of the current audio frame to be output, so as to dynamically adjust the output gain, thereby making the output of the audio frame to be output more natural.

[0098] In some optional implementations of the embodiments of the present disclosure, the output gain of the audio frame to be output is determined based on the audio type of the audio frame to be output, including: in a case where the audio type of the audio frame to be output is a speech type, determining the output gain of the audio frame to be output as a first preset value; and in a case where the audio type of the audio frame to be output is a noise type, determining the output gain of the audio frame to be output as a second preset value, wherein the first preset value is greater than the second preset value.

[0099] For example, the first preset value can be 1, and the second preset value can be 0. In a case where the audio type of the audio frame to be output is a speech type, the output gain of the audio frame to be output can be set to 1, and the audio frame is normally output. In a case where the audio type of the audio frame to be output is a noise type, the output gain of the audio frame to be output can be set to 0, and the audio frame is eliminated to achieve the purpose of noise reduction. It should be noted that the first preset value and the second preset value are only examples, and the first preset value and the second preset value can also be other values, such as 0.8 and 0.2, and are not limited.

[0100] In some optional implementations of the embodiments of the present disclosure, the output gain of the audio frame to be output is determined based on the audio type of the audio frame to be output, and further includes: in a case where the audio type of the audio frame to be output is a noise type, and a preset number of frames of the audio frames that have been output include an audio frame of a speech type, performing smoothing processing based on the output gain of the last audio frame to be output to obtain the output gain of the audio frame to be output.

[0101] The smoothing processing can obtain a gradually changing output gain, which can prevent the audio type from repeatedly jumping, remove the burr part in the audio, and make the output of the audio frame to be output more natural, compared with the method of setting the fixed first preset value and the second preset value.

[0102] For example, the preset number of frames can be N, that is, in a case where the audio type of the audio frame to be output is the noise type and N audio frames of the speech type have been output before the audio frame to be output, the output gain of the audio frame to be output is determined according to the output gain of the last output audio frame, and optionally, the output gain of the audio frame to be output is obtained by gradually decreasing the output gain of the last output audio frame, for example, the output gain of the last output audio frame is 0.8, the output gain of the audio frame to be output is obtained by subtracting a gain adjustment value, for example, 0.2, from the output gain of the last output audio frame, that is, 0.6. It should be noted that the values of the output gain and the decreasing values are only illustrative and are not limited.

[0103] Further, the obtained output gain can be used to process the audio frame to be output to obtain an output audio frame, which can refer to an audio frame that needs to be output and played, and the output manner can include but is not limited to direct output of the current device or transmission to other devices for output through wired or wireless communication.

[0104] For example, the output gain can be multiplied by a parameter value corresponding to the audio frame to be output to realize noise reduction processing on the audio frame to be output, for example, if the output gain is 1, the audio frame to be output remains unchanged, and if the output gain is 0, the audio frame to be output is set to zero to remove the noise frame.

[0105] The technical scheme of the embodiment of the present disclosure determines the output gain of the audio frame to be output according to the audio type of the audio frame to be output, and the output gain can be used to adjust the output state of the audio frame to be output to realize noise reduction processing and improve the quality of the output audio frame.

[0106] Figure 5 is a structural schematic diagram of an audio processing apparatus provided by the embodiment of the present disclosure. As shown in Figure 5 The apparatus comprises:

[0107] The type determination module 510 is configured to acquire an audio frame to be processed, and determine an audio type of the audio frame based on a current recognition threshold.

[0108] The state determination module 520 is configured to determine a determination state of the recognized audio type based on feature information of the recognized continuous audio frame in a case where the current audio frame meets a threshold adjustment condition.

[0109] The threshold adjustment module 530 is configured to adjust the current recognition threshold according to the determination state, and the adjusted recognition threshold is used for audio type recognition on a next audio frame.

[0110] In some optional implementation manners of the embodiment of the present disclosure, the audio type includes a speech type and a noise type.

[0111] The threshold adjustment condition is that the audio type of the current audio frame is a noise type and the audio type of the previous audio frame is a speech type.

[0112] In some optional implementations of the embodiments of the present disclosure, the state determination module 520 is further configured to:

[0113] determine feature information of a continuous speech frame before the current audio frame, and compare the feature information with a judgment threshold of the feature information;

[0114] determine the determination state of the recognized audio type based on a comparison result.

[0115] In some optional implementations of the embodiments of the present disclosure, the feature information includes one or more of the following: a length of the continuous speech frame, a recognition probability, a pitch frequency, and an energy value.

[0116] In some optional implementations of the embodiments of the present disclosure, the determination state includes an error state and a correct state; and the threshold adjustment module 530 is further configured to:

[0117] increase the current recognition threshold when the determination state is the error state;

[0118] decrease the current recognition threshold when the determination state is the correct state.

[0119] In some optional implementations of the embodiments of the present disclosure, after the audio frame to be processed is obtained, the audio frame is added to a cache area, where the cache area is used to store a plurality of audio frames that are not output, the current audio frame is located at the last frame of the cache area, and a first frame in the cache area is an output frame.

[0120] In some optional implementations of the embodiments of the present disclosure, the apparatus is further configured to:

[0121] if the audio type of the current audio frame is a speech type, set the audio type of each audio frame in the cache area as the speech type;

[0122] if the audio type of the current audio frame is a noise type, set the audio type of the last audio frame in the cache area as the noise type.

[0123] In some optional implementations of the embodiments of the present disclosure, the apparatus further includes:

[0124] determine a current threshold range in which the adjusted recognition threshold is located, and determine a cache area length according to the current threshold range.

[0125] In some optional implementations of the embodiments of the present disclosure, the apparatus further includes:

[0126] In a case where the audio types of the audio frames in the buffer area are all noise types, the buffer area is emptied, and a buffer area is reconstructed based on a current buffer area length.

[0127] In some optional implementations of the embodiments of the present disclosure, the apparatus further includes:

[0128] a gain determination module configured to determine an output gain of the audio frame to be output based on an audio type of the audio frame to be output;

[0129] an audio output module configured to process the audio frame to be output based on the output gain to obtain an output audio frame, and output the output audio frame.

[0130] In some optional implementations of the embodiments of the present disclosure, the gain determination module is further configured to:

[0131] In a case where the audio type of the audio frame to be output is a speech type, the output gain of the audio frame to be output is determined as a first preset value;

[0132] In a case where the audio type of the audio frame to be output is a noise type, the output gain of the audio frame to be output is determined as a second preset value, and the first preset value is greater than the second preset value.

[0133] In some optional implementations of the embodiments of the present disclosure, the gain determination module is further configured to:

[0134] In a case where the audio type of the audio frame to be output is a noise type, and a preset number of output audio frames include an audio frame of a speech type, the output gain of the audio frame to be output is obtained by performing smoothing processing based on an output gain of a last audio frame to be output.

[0135] In some optional implementations of the embodiments of the present disclosure, the type determination module 510 is further configured to:

[0136] extract an audio feature of the audio frame, input the audio feature into an audio recognition model, and obtain a recognition probability of the audio frame;

[0137] determine the audio type of the audio frame based on the current recognition threshold and the recognition probability.

[0138] In some optional implementations of the embodiments of the present disclosure, the training apparatus of the audio recognition model includes:

[0139] a label setting module configured to obtain a noise-free audio, and set a label for each audio segment in the noise-free audio;

[0140] a sample making module configured to obtain noise information, and superimpose the noise information onto the noise-free audio to form a sample audio, wherein the noise information comprises at least one of steady-state noise, transient noise and howling noise;

[0141] a model training module configured to iteratively train the audio recognition model to be trained based on the sample audio until a trained audio recognition model is obtained.

[0142] In some optional implementations of the embodiments of the present disclosure, the training apparatus of the audio recognition model can be further configured to:

[0143] adjust a signal-to-noise ratio in the sample audio; and / or,

[0144] filter the sample audio based on a preset filter.

[0145] The audio processing apparatus provided by the embodiments of the present disclosure can execute the audio processing method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0146] It should be noted that each unit and module included in the apparatus is only divided according to the function logic, and is not limited to the above division, as long as the corresponding function can be implemented; in addition, the specific name of each functional unit is only for convenient distinction, and does not limit the protection scope of the embodiments of the present disclosure.

[0147] Reference will be made to the following description of the embodiments of the present disclosure in conjunction with the accompanying drawings. Figure 6 which shows a structural schematic diagram of an electronic device (for example, a terminal device or a server in Figure 6 The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (for example, vehicle-mounted navigation terminals) and the like, and fixed terminals such as digital TVs, desktop computers and the like. Figure 6 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0148] As Figure 6As shown, the electronic device 400 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. Various programs and data required for the operation of the electronic device 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0149] Generally, the following devices can be connected to the I / O interface 405: input devices 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 408 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 409. The communication devices 409 can allow the electronic device 400 to communicate wirelessly or wired with other devices to exchange data. Although Figure 6 The electronic device 400 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0150] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 409, or installed from the storage devices 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0151] The electronic device provided by embodiments of the present disclosure and the audio processing method provided by the above-mentioned embodiments belong to the same inventive concept, and technical details not described in detail in the present embodiments can be referred to the above-mentioned embodiments, and the present embodiments have the same beneficial effects as the above-mentioned embodiments.

[0152] The present embodiments provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio processing method provided by the above-mentioned embodiments.

[0153] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, in which the computer-readable program code is contained. Such a propagated data signal can take any of a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the foregoing. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including, but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0154] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0155] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device described above, and can be accessed via the electronic device described above.

[0156] The computer-readable medium described above carries one or more programs, which, when executed by the electronic device described above, cause the electronic device to:

[0157] obtain an audio frame to be processed, determine an audio type of the audio frame based on a current recognition threshold;

[0158] in a case where the current audio frame satisfies a threshold adjustment condition, determine a determination state of the recognized audio type based on feature information of the recognized continuous audio frame;

[0159] adjust the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for audio type recognition of a next audio frame.

[0160] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ as well as conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0161] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0162] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit / module does not constitute a limitation on the unit itself.

[0163] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0164] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0165] According to one or more embodiments of the present disclosure, Example One provides an audio processing method, comprising:

[0166] obtaining an audio frame to be processed, determining an audio type of the audio frame based on a current recognition threshold;

[0167] in a case where the current audio frame meets a threshold adjustment condition, determining a determination state of the recognized audio type based on feature information of the recognized continuous audio frame;

[0168] adjusting the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for audio type recognition of a next audio frame.

[0169] According to one or more embodiments of the present disclosure, Example Two provides an audio processing method, further comprising:

[0170] The audio type includes a speech type and a noise type;

[0171] The threshold adjustment condition is that the audio type of the current audio frame is the noise type and the audio type of a previous audio frame is the speech type.

[0172] According to one or more embodiments of the present disclosure, Example Three provides an audio processing method, further comprising:

[0173] The determining state of the identified audio type based on the feature information of the identified continuous audio frame includes:

[0174] The feature information of the continuous speech frame before the current audio frame is determined, and the feature information is compared with a judgment threshold of the feature information;

[0175] The determining state of the identified audio type is determined based on the comparison result.

[0176] According to one or more embodiments of the present disclosure, Example Four provides an audio processing method, further comprising:

[0177] The feature information includes one or more of the length, recognition probability, fundamental frequency and energy value of the continuous speech frame.

[0178] According to one or more embodiments of the present disclosure, Example Five provides an audio processing method, further comprising:

[0179] The determining state includes an error state and a correct state;

[0180] The current recognition threshold is adjusted according to the determining state, including:

[0181] In the case that the determining state is the error state, the current recognition threshold is increased;

[0182] In the case that the determining state is the correct state, the current recognition threshold is decreased.

[0183] According to one or more embodiments of the present disclosure, Example Six provides an audio processing method, further comprising:

[0184] After obtaining the audio frame to be processed, the audio frame is added to a cache area, wherein the cache area is used to store a plurality of audio frames that have not been output, the current audio frame is located at the last frame of the cache area, and the first frame in the cache area is an output frame.

[0185] According to one or more embodiments of the present disclosure, Example Seven provides an audio processing method, further comprising:

[0186] After determining the audio type of the audio frame based on the current recognition threshold, the method further comprises:

[0187] If the audio type of the current audio frame is a speech type, the audio types of the audio frames in the cache area are set to the speech type;

[0188] If the audio type of the current audio frame is a noise type, the audio type of the last audio frame in the cache area is set to the noise type.

[0189] According to one or more embodiments of the present disclosure, Example Eight provides an audio processing method, further comprising:

[0190] After adjusting the current recognition threshold according to the determination state, the method further comprises:

[0191] Determining a current threshold range in which the adjusted recognition threshold is located, and determining a buffer length according to the current threshold range.

[0192] According to one or more embodiments of the present disclosure, Example Nine provides an audio processing method, comprising:

[0193] In a case where the audio type of each audio frame in the buffer is a noise type, emptying the buffer and rebuilding a buffer based on a current buffer length.

[0194] According to one or more embodiments of the present disclosure, Example Ten provides an audio processing method, further comprising:

[0195] Determining an output gain of the audio frame to be output based on the audio type of the audio frame to be output;

[0196] Processing the audio frame to be output based on the output gain to obtain an output audio frame, and outputting.

[0197] According to one or more embodiments of the present disclosure, Example Eleven provides an audio processing method, further comprising:

[0198] The determining of the output gain of the audio frame to be output based on the audio type of the audio frame to be output comprises:

[0199] In a case where the audio type of the audio frame to be output is a speech type, determining the output gain of the audio frame to be output as a first preset value;

[0200] In a case where the audio type of the audio frame to be output is a noise type, determining the output gain of the audio frame to be output as a second preset value, wherein the first preset value is greater than the second preset value.

[0201] According to one or more embodiments of the present disclosure, Example Twelve provides an audio processing method, further comprising:

[0202] The determining of the output gain of the audio frame to be output based on the audio type of the audio frame to be output further comprises:

[0203] In a case where the audio type of the audio frame to be output is a noise type and a preset number of frames of the output audio frames include an audio frame of a speech type, performing smoothing processing based on an output gain of a last audio frame to be output to obtain an output gain of the audio frame to be output.

[0204] According to one or more embodiments of the present disclosure, Example Thirteen provides an audio processing method, further comprising:

[0205] The determining the audio type of the audio frame based on the current recognition threshold comprises:

[0206] extracting an audio feature of the audio frame, inputting the audio feature into an audio recognition model to obtain a recognition probability of the audio frame;

[0207] determining the audio type of the audio frame based on the current recognition threshold and the recognition probability.

[0208] According to one or more embodiments of the present disclosure, Example Fourteen provides an audio processing method, further comprising:

[0209] The training method of the audio recognition model comprises:

[0210] obtaining noise-free audio, setting a label for each audio segment in the noise-free audio;

[0211] obtaining noise information, superimposing the noise information into the noise-free audio to form a sample audio, wherein the noise information comprises at least one of a steady-state noise, a transient noise and a howling noise;

[0212] iteratively training the audio recognition model to be trained based on the sample audio until a trained audio recognition model is obtained.

[0213] According to one or more embodiments of the present disclosure, Example Fifteen provides an audio processing method, further comprising:

[0214] The method further comprises:

[0215] adjusting a signal-to-noise ratio in the sample audio; and / or,

[0216] filtering processing the sample audio based on a preset filter.

[0217] According to one or more embodiments of the present disclosure, Example Sixteen provides an audio processing device, comprising:

[0218] a type determination module configured to obtain an audio frame to be processed, and determine an audio type of the audio frame based on a current recognition threshold;

[0219] The status determination module is used to determine the determination status of the identified audio type based on the feature information of the identified consecutive audio frames when the current audio frame meets the threshold adjustment conditions.

[0220] The threshold adjustment module is used to adjust the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used to identify the audio type of the next audio frame.

[0221] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0222] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0223] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An audio processing method, characterized by, The method comprises: obtaining an audio frame to be processed, and determining an audio type of the audio frame based on a current recognition threshold; the audio type comprises a speech type and a noise type; in a case where a current audio frame meets a threshold adjustment condition, determining a determination state of the recognized audio type based on feature information of the recognized continuous audio frame; the determination state comprises an error state and a correct state; the threshold adjustment condition is that the audio type of the current audio frame is the noise type and the audio type of a previous audio frame is the speech type; adjusting the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for audio type recognition of a next audio frame.

2. The method of claim 1, wherein, The method further comprises: determining feature information of the continuous speech frame before the current audio frame, and comparing the feature information with a judgment threshold of the feature information; determining the determination state of the recognized audio type based on a comparison result.

3. The method of claim 2, wherein, The feature information comprises one or more of the following: length, recognition probability, pitch frequency and energy value of the continuous speech frame.

4. The method of claim 1, wherein, The method further comprises: in a case where the determination state is the error state, increasing the current recognition threshold; in a case where the determination state is the correct state, decreasing the current recognition threshold.

5. The method of claim 1, wherein, After obtaining the audio frame to be processed, the audio frame is added to a cache area, wherein the cache area is used to store a plurality of audio frames that are not output, the current audio frame is located at the last frame of the cache area, and a first frame in the cache area is an output frame.

6. The method of claim 5, wherein, After determining the audio type of the audio frame based on the current recognition threshold, the method further comprises: if the audio type of the current audio frame is the speech type, setting the audio type of each audio frame in the cache area as the speech type; if the audio type of the current audio frame is the noise type, setting the audio type of the last audio frame in the cache area as the noise type.

7. The method of claim 5, wherein, After adjusting the current recognition threshold according to the determination state, the method further comprises: determining a current threshold range in which the adjusted recognition threshold is located, and determining a cache area length according to the current threshold range.

8. The method of claim 5, wherein, The method further comprises: in a case where the audio type of each audio frame in the cache area is the noise type, emptying the cache area, and rebuilding a cache area based on a current cache area length.

9. The method of claim 1, wherein, The method further comprises: determining an output gain of the audio frame to be output based on the audio type of the audio frame to be output; processing the audio frame to be output based on the output gain to obtain an output audio frame, and outputting the output audio frame.

10. The method of claim 9, wherein, The method further comprises: in a case where the audio type of the audio frame to be output is the speech type, determining the output gain of the audio frame to be output as a first preset value; in a case where the audio type of the audio frame to be output is the noise type, determining the output gain of the audio frame to be output as a second preset value, wherein the first preset value is greater than the second preset value.

11. The method of claim 10, wherein, The output gain of the audio frame to be output is determined based on an audio type of the audio frame to be output, and the method further includes: In a case where the audio type of the audio frame to be output is a noise type and a preset number of frames of the audio frames that have been output include an audio frame of a speech type, performing smoothing processing on an output gain of a last audio frame to be output to obtain the output gain of the audio frame to be output.

12. The method of claim 1, wherein, The determining of the audio type of the audio frame based on the current recognition threshold includes: extracting an audio feature of the audio frame, inputting the audio feature into an audio recognition model to obtain a recognition probability of the audio frame; determining the audio type of the audio frame based on the current recognition threshold and the recognition probability.

13. The method of claim 12, wherein, The training method of the audio recognition model includes: obtaining noise-free audio, setting a label for each audio segment in the noise-free audio; obtaining noise information, superimposing the noise information on the noise-free audio to form a sample audio, wherein the noise information includes at least one of a steady-state noise, a transient noise and a howling noise; iteratively training a to-be-trained audio recognition model based on the sample audio until a trained audio recognition model is obtained.

14. The method of claim 13, wherein, The method further includes: adjusting a signal-to-noise ratio in the sample audio; and / or, performing filtering processing on the sample audio based on a preset filter.

15. An audio processing device, comprising: includes: a type determination module configured to obtain an audio frame to be processed and determine an audio type of the audio frame based on a current recognition threshold; the audio type includes a speech type and a noise type; a state determination module configured to determine a determination state of a recognized audio type based on feature information of the recognized continuous audio frames in a case where a current audio frame meets a threshold adjustment condition, the determination state including an error state and a correct state; the threshold adjustment condition is that the audio type of the current audio frame is a noise type and the audio type of a previous audio frame is a speech type; a threshold adjustment module configured to adjust the current recognition threshold according to the determination state, wherein the adjusted recognition threshold is used for recognizing the audio type of a next audio frame.

16. An electronic device, comprising: The electronic device includes: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method of any one of claims 1-14.

17. A storage medium containing computer executable instructions for performing the audio processing method of any one of claims 1-14 when executed by a computer processor.

Images