Audio processing method, video processing method and device, medium and electronic device

Abstract

The present disclosure provides an audio processing method, a video processing method, an audio processing device, a video processing device, a computer readable storage medium and an electronic device, and relates to the technical field of audio processing. The audio processing method comprises: performing an initial feature extraction operation on audio data to obtain first audio features; performing a processing procedure based on feature splitting on a time domain on the first audio features to obtain second audio features; enhancing the difference between speech features and non-speech features in the second audio features based on an attention mechanism to obtain third audio features; and performing a fully connected mapping operation on the third audio features to obtain a speech detection result of the audio data. The present disclosure can improve the accuracy of speech detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of audio processing, and in particular, to an audio processing method, a video processing method, an audio processing apparatus, a video processing apparatus, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] With the development of terminal technology, the demand of users for albums is no longer limited to reviewing photos and videos, and the demand for multi-modal intelligent editing and creation is also increasing. Under this background, voice activity detection (VAD) technology plays an important role. As a preprocessing technology for audio intelligent processing, it can automatically divide the frames of audio signals into speech and non-speech. In addition, the effects of speech recognition, voice detection, and voice separation technologies also depend on the processing results of voice activity detection.

[0003] Currently, voice activity detection may have the problems of low detection accuracy and poor speech recognition effect. SUMMARY

[0004] The present disclosure provides an audio processing method, a video processing method, an audio processing apparatus, a video processing apparatus, a computer readable storage medium, and an electronic device, thereby at least partially overcoming the problem of low voice activity detection accuracy.

[0005] According to a first aspect of the present disclosure, an audio processing method is provided, comprising: performing an initial feature extraction operation on audio data to obtain first audio features; performing a processing procedure based on feature splitting on a time domain on the first audio features to obtain second audio features; enhancing the difference between speech features and non-speech features in the second audio features based on an attention mechanism to obtain third audio features; and performing a fully connected mapping operation on the third audio features to obtain a speech detection result of the audio data.

[0006] According to a second aspect of the present disclosure, a video processing method is provided, comprising: obtaining a video and extracting audio data in the video; and performing speech detection on the audio data using the above-mentioned audio processing method to determine a speech segment in the video.

[0007] According to a third aspect of the present disclosure, an audio processing apparatus is provided, comprising: a first feature determination module configured to perform an initial feature extraction operation on audio data to obtain first audio features; a second feature determination module configured to perform a feature splitting based processing procedure on the first audio features to obtain second audio features; a third feature determination module configured to enhance differences between speech features and non-speech features in the second audio features based on an attention mechanism to obtain third audio features; and a speech detection module configured to perform a fully connected mapping operation on the third audio features to obtain a speech detection result of the audio data.

[0008] According to a fourth aspect of the present disclosure, a video processing apparatus is provided, comprising: an audio extraction module configured to obtain a video and extract audio data in the video; and a segment determination module configured to perform speech detection on the audio data using the audio processing method to determine a speech segment in the video.

[0009] According to a fifth aspect of the present disclosure, a computer readable storage medium is provided, having a computer program stored thereon, the program being executed by a processor to implement the audio processing method or the video processing method.

[0010] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory configured to store one or more programs, the one or more programs being executed by the processor to implement the audio processing method or the video processing method.

[0011] In the technical solutions provided by some embodiments of the present disclosure, a first audio feature extracted from audio data is processed based on a feature splitting procedure in a time domain to obtain a second audio feature, differences between speech features and non-speech features are enhanced based on an attention mechanism to obtain a third audio feature, and a fully connected mapping operation is performed on the third audio feature to obtain a speech detection result of the audio data. On the one hand, by using the feature splitting procedure in the time domain, multi-level features in the time domain can be effectively determined, which helps to improve the accuracy of speech detection. On the other hand, the present disclosure combines the attention mechanism to enhance the differences between speech features and non-speech features, which provides a good data basis for subsequent classification operations and can further improve the accuracy of speech detection.

[0012] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure. It is to be understood that the drawings are designed solely for purposes of illustration to be used in conjunction with the description in specifying the present embodiments. As such, other drawings, having different numbers and labels from those of the drawing figures being discussed, can represent other embodiments consistent with the present disclosure, which can be understood clearly and entirely by those skilled in the art from a study of the drawings, specifications, and concordantly the base disclosure.

[0014] Figure 1 A schematic diagram of an exemplary system architecture of an audio processing scheme of an embodiment of the present disclosure is shown.

[0015] Figure 2 An input-output schematic diagram of an audio processing scheme of an embodiment of the present disclosure is shown.

[0016] Figure 3 A flowchart of an audio processing method according to an exemplary embodiment of the present disclosure is shown schematically.

[0017] Figure 4 A schematic diagram of correcting a speech detection result of an embodiment of the present disclosure is shown.

[0018] Figure 5 A model architecture diagram of an audio processing model of an embodiment of the present disclosure is shown schematically.

[0019] Figure 6 A model architecture diagram of a time-domain feature split processing unit corresponding to a time-domain feature split sub-processing process of an embodiment of the present disclosure is shown schematically.

[0020] Figure 7 A schematic diagram of a generation process of training data for training an audio processing model of an embodiment of the present disclosure is shown.

[0021] Figure 8 A flowchart of a video processing method according to an exemplary embodiment of the present disclosure is shown schematically.

[0022] Figure 9 A flowchart of constructing a wonderful childhood memory by applying an exemplary scheme of the present disclosure is shown schematically.

[0023] Figure 10 A block diagram of an audio processing apparatus according to an exemplary embodiment of the present disclosure is shown schematically.

[0024] Figure 11 A block diagram of an audio processing apparatus according to another exemplary embodiment of the present disclosure is shown schematically.

[0025] Figure 12 A block diagram of a video processing apparatus according to an exemplary embodiment of the present disclosure is shown schematically.

[0026] Figure 13A block diagram of an electronic device according to an example embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Example embodiments now will be described more fully hereinafter with reference to the accompanying drawings. Example embodiments, however, can be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures have not been described in detail so as not to obscure aspects of the disclosure.

[0028] In addition, the drawings of the present disclosure are only schematic and are not necessarily drawn to scale. The same reference numerals in different drawings represent the same or similar functionality throughout the several drawings; thus, repeated description is omitted. Some of the drawings can be schematic or exaggerated representations of concepts as described herein for purposes of explanation. The drawings are provided to facilitate understanding of the present disclosure. Thus, the present disclosure should not be construed as limited to the embodiments set forth herein; many embodiments can be made without departing from the spirit and scope of the present disclosure. Like reference numerals in different drawings represent the same or similar elements.

[0029] The flow diagrams in the drawings are merely examples and do not have to include all of the steps. For example, some steps can be combined, some steps can be omitted, and some steps can be split into multiple steps, so the actual execution order can be changed according to actual conditions. In addition, all the terms "first", "second", "third", "fourth", and the like in the following are only for the purpose of distinction, and should not be regarded as a limitation of the present disclosure.

[0030] In some model-based audio scenarios, due to the diversity of audio scenarios and the possibility of low signal-to-noise ratio, the robustness of the current model is insufficient. In addition, for long-time audio data, the modeling capability of the existing model (such as a convolutional neural network) is difficult to achieve ideal results, causing the problem of poor voice detection effect of audio data.

[0031] In order to at least improve the voice detection accuracy of audio data, the present disclosure provides a new audio processing scheme. Unless otherwise specified, the voice detection referred to in the embodiments of the present disclosure refers to active voice detection, that is, detecting the voice part and non-voice part in the audio data.

[0032] Figure 1 A schematic diagram of an exemplary system architecture of an audio processing scheme of an embodiment of the present disclosure is shown. As shown, the system architecture can include a terminal device 110 and a server 120. The terminal device 110 can interact with the server 120 through a network, which can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc. Figure 1

[0033] The terminal device 110 can include, but is not limited to, a smartphone, a tablet computer, a smart wearable device, a personal computer, etc., and the present disclosure does not limit the type of the terminal device 110. The server 120 can be a single server, or a server cluster composed of multiple servers, etc.

[0034] The audio processing scheme of an embodiment of the present disclosure can be executed by the terminal device 110, executed by the server 120, or jointly executed by the terminal device 110 and the server 120.

[0035] In an example where the audio processing scheme of an embodiment of the present disclosure is executed by the terminal device 110, the terminal device 110 can analyze and process audio data to determine a speech detection result corresponding to the audio data.

[0036] In an example where the audio processing scheme of an embodiment of the present disclosure is executed by the server 120, the terminal device 110 can send audio data to the server 120, and the server 120 can analyze and process the audio data to determine a speech detection result corresponding to the audio data.

[0037] The audio processing scheme of an embodiment of the present disclosure can be implemented based on an audio processing model. In this case, the server 120 can train the model using training data to obtain a trained audio processing model. Subsequently, the server 120 can distribute the trained audio processing model to the terminal device 110, and thus the terminal device 110 can process audio data currently to be analyzed using the trained audio processing model to determine a speech detection result corresponding to the audio data.

[0038] In addition, the server 120 can also directly use the trained audio processing model for inference to determine a speech detection result corresponding to the audio data.

[0039] To improve the speech detection effect, the server 120 can preprocess the training data, such as adding noise and / or reverberation, and performing spectral enhancement, thereby improving the robustness of the model and making the detection of speech more accurate.

[0040] For convenience of description, the following will be described by taking an example of the terminal device executing the audio processing scheme of an embodiment of the present disclosure. ​

[0041] Figure 2 An input-output diagram of an audio processing scheme of an embodiment of the present disclosure is shown. Referring to Figure 2 The input of the audio processing method of the present disclosure is audio data, and the output is a speech detection result.

[0042] First, the terminal device can obtain audio data, and perform an initial feature extraction operation on the audio data to obtain first audio features. Next, the terminal device can perform a processing procedure based on feature splitting on a time domain on the first audio features to obtain second audio features. Subsequently, the terminal device can enhance the difference between speech features and non-speech features in the second audio features based on an attention mechanism to obtain third audio features. Then, the terminal device can perform a fully connected mapping operation on the third audio features to obtain a speech detection result of the audio data.

[0043] As Figure 2 is exemplarily shown, it should be noted that the speech detection result is for each frame of the audio data, for example, an audio frame containing speech is recorded as 1, and an audio frame containing non-speech is recorded as 0; or an audio frame containing non-speech is recorded as 1, and an audio frame containing speech is recorded as 0.

[0044] The audio processing method of the present embodiment will be described below.

[0045] Figure 3 A flowchart of the audio processing method of an exemplary embodiment of the present disclosure is shown. Referring to Figure 3 The audio processing method can include the following steps:

[0046] S32. Perform an initial feature extraction operation on the audio data to obtain first audio features.

[0047] In an exemplary embodiment of the present disclosure, the audio data is data that is currently to be analyzed for speech detection. The present disclosure does not limit the source of the audio data. For example, the audio data can be audio data obtained by the terminal device recording the environment in which it is located. For another example, the audio data can also be audio data generated by the terminal device performing an audio extraction operation on a video, which can be a video in the photo album of the terminal device. For yet another example, the audio data can be software-synthesized data.

[0048] The audio data can be generated by the terminal device. In addition, the terminal device can also obtain the audio data from other devices.

[0049] After the audio data is determined, the terminal device can perform an initial feature extraction operation on the audio data to obtain first audio features.

[0050] According to some embodiments of the present disclosure, the terminal device can extract Fbank (Filter Bank) features of the audio data. The Fbank features can be extracted in a manner consistent with the principles of human hearing, for example, by pre-emphasis, framing, windowing, short-time Fourier transform, Mel filtering, etc., and the present disclosure does not limit this.

[0051] In addition, the present disclosure does not limit the dimensions, frame length, frame shift, etc. of the extracted features. For example, 80-dimensional Fbank features with a frame length of 25 ms and a frame shift of 10 ms can be extracted.

[0052] The terminal device can determine the extracted Fbank features as the first audio features.

[0053] It should be understood that the Fbank features are only exemplary descriptions, and the first audio features of the present disclosure can also be MFCC (Mel-scale Frequency Cepstral Coefficients) features, PNCC (Power-Normalized Cepstral Coefficients) features, etc.

[0054] According to some other embodiments of the present disclosure, first, the terminal device can perform an initial feature extraction operation on the audio data to obtain initial features. The initial features can be, for example, the Fbank features, MFCC features, or PNCC features described above, etc.

[0055] Next, the terminal device can perform convolution processing on the initial features to obtain the first audio features. In the embodiments of the present disclosure, the convolution processing can include a combination of a one-dimensional convolution operation (Conv1d), a linear rectification operation (Relu), and a batch normalization operation (BN).

[0056] It can be understood that the first audio features referred to in the present disclosure are usually sequences of audio features.

[0057] S34. Perform a processing procedure based on feature splitting in the time domain on the first audio features to obtain second audio features.

[0058] In exemplary embodiments of the present disclosure, the processing procedure based on feature splitting in the time domain can refer to a feature splitting operation in the time sequence included in the processing procedure of the first audio features.

[0059] The processing procedure based on feature splitting in time domain can include at least one level of time-domain feature splitting sub-procedure. That is, the terminal device can perform at least one level of time-domain feature splitting sub-procedure on the first audio feature, and generate the second audio feature based on the output result of the at least one level of time-domain feature splitting sub-procedure.

[0060] According to some embodiments of the present disclosure, the terminal device performs more than two levels of time-domain feature splitting sub-procedures on the first audio feature, and the sub-procedures are configured in a cascaded manner in algorithm, that is, the output of each level of time-domain feature splitting sub-procedure is the input of the next level of time-domain feature splitting sub-procedure. In addition, the algorithm execution process of each level of time-domain feature splitting sub-procedure can be the same or different, but each level of time-domain feature splitting sub-procedure includes a feature splitting operation in time sequence.

[0061] In this case, the terminal device can fuse the output results of the levels of time-domain feature splitting sub-procedures, and determine the second audio feature according to the fused result. In this way, the fusion of multi-layer features of different levels is realized, and the accuracy of the detection result can be improved.

[0062] The time-domain feature splitting sub-procedure of the embodiments of the present disclosure is exemplarily described below.

[0063] Firstly, the terminal device can perform a feature splitting operation in time domain on the audio feature input into the time-domain feature splitting sub-procedure to obtain a plurality of first intermediate features. In order to ensure the stability of the algorithm processing, the audio feature can be split at equal intervals in time sequence, that is, the time length of each first intermediate feature is consistent.

[0064] According to some embodiments of the present disclosure, before the feature splitting operation, the audio feature can also be subjected to convolution processing. That is, the terminal device can perform convolution processing on the audio feature input into the time-domain feature splitting sub-procedure, and perform a feature splitting operation in time domain on the feature obtained after the convolution processing to obtain a plurality of first intermediate features.

[0065] Next, the terminal device can perform convolution operation on the plurality of first intermediate features to obtain a plurality of second intermediate features. The second intermediate features can correspond to the first intermediate features one by one.

[0066] Specifically, the terminal device can take the first first intermediate feature in the time domain as the first second intermediate feature. The terminal device can perform convolution processing on the second first intermediate feature in the time domain to obtain the second second intermediate feature. The terminal device can add the i-th first intermediate feature in the time domain and the (i-1)-th second intermediate feature, and perform convolution processing on the added features to obtain the i-th second intermediate feature. Wherein, i is a positive integer greater than or equal to 3.

[0067] Subsequently, the terminal device can splice the obtained multiple second intermediate features to obtain a third intermediate feature.

[0068] Then, the terminal device can perform feature enhancement on the third intermediate feature to obtain the audio feature output by the time-domain feature splitting sub-process.

[0069] According to some embodiments of the present disclosure, the terminal device can directly output the feature after feature enhancement.

[0070] According to other embodiments of the present disclosure, the terminal device can perform feature enhancement on the third intermediate feature to obtain a fourth intermediate feature, and add the fourth intermediate feature to the audio feature input into the time-domain feature splitting sub-process to obtain the audio feature output by the time-domain feature splitting sub-process. This process can be implemented by a model, and thus, by adding the input and the output, the problem of model degradation can be solved, which helps the model to converge quickly.

[0071] In addition, before feature enhancement, the terminal device can first perform convolution processing on the third intermediate feature, and perform feature enhancement on the feature obtained after convolution processing to obtain a fourth intermediate feature.

[0072] The present disclosure does not limit the specific way of feature enhancement, for example, the terminal device can determine the importance of each feature channel in the third intermediate feature, and configure a weight for each feature channel according to the importance of each feature channel. In this way, the features useful for the audio processing task can be enhanced and the features useless or less useful for the audio processing task can be suppressed.

[0073] It should be noted that the present disclosure does not limit the number of time-domain feature splitting sub-processes, which can be related to the processing capability of the terminal device and the complexity of the audio processing task.

[0074] S36. Enhance the difference between the speech feature and the non-speech feature in the second audio feature based on the attention mechanism to obtain a third audio feature.

[0075] The attention mechanism configured in the embodiments of the present disclosure can enhance the difference between the speech feature and the non-speech feature. Specifically, the attention conversion process of the present disclosure can be implemented in combination with the Encoder-Decoder framework, and the present disclosure does not limit this.

[0076] S38. Perform a fully connected mapping operation on the third audio feature to obtain the speech detection result of the audio data.

[0077] In an exemplary embodiment of this disclosure, the obtained speech detection result of the audio data includes the speech detection result of each audio frame in the audio data. The speech detection result characterizes whether the audio frame is speech or non-speech. For example, audio frames containing speech are recorded as 1, and audio frames containing non-speech are recorded as 0; or, audio frames containing non-speech are recorded as 1, and audio frames containing speech are recorded as 0.

[0078] Since the speech detection results are obtained for each audio frame, a speech segment may be discretely divided, which may lead to unevenness in the speech detection results and affect the accuracy of the speech detection results.

[0079] To address this issue, this disclosure also provides a smoothing method.

[0080] First, the terminal device performs a fully connected mapping operation on the third audio feature to determine the first speech detection result for each audio frame in the audio data. Next, the terminal device can use a first frame count threshold and a stacked frame count threshold to correct the first speech detection result to determine the second speech detection result for each audio frame in the audio data, and output the second speech detection result as the final detection result.

[0081] The first frame count threshold is the frame count requirement set for the transition from voice state to non-voice state, and the second frame count threshold is the frame count requirement set for the transition from non-voice state to voice state. It can be understood that both the first and second frame count thresholds represent the minimum required number of frames. The first and second frame count thresholds can be the same or configured to be different.

[0082] Figure 4 A schematic diagram illustrating the correction of speech detection results according to an embodiment of the present disclosure is shown.

[0083] by Figure 4 For example, if non-speech is recorded as 0 and speech as 1, and both the first frame threshold and the second frame threshold are configured as 3, then the effect of converting the first speech detection result into the second speech detection result can be achieved.

[0084] The accuracy of the test can be improved by smoothing the speech detection results based on the frame number threshold disclosed in this disclosure.

[0085] In addition to the smoothing method based on frame number thresholds mentioned above, smoothing can also be performed by, for example, median filtering, and this disclosure does not limit this.

[0086] Based on the above exemplary audio processing method of the present disclosure, accurate activity speech detection can be achieved, and high-quality data basis can be provided for subsequent algorithms such as speech recognition and voice segmentation.

[0087] The present disclosure can implement part of the above audio processing method through an audio processing model. The audio processing model adopted by the present disclosure is described below.

[0088] The audio processing module of the present disclosure embodiment can include a time domain feature splitting processing module, an attention layer, and a fully connected layer. Specifically, the time domain feature splitting processing module can be used to perform a time domain feature splitting-based processing process on the first audio feature to obtain the second audio feature. The attention layer can be used to enhance the difference between the speech feature and the non-speech feature in the second audio feature based on an attention mechanism to obtain the third audio feature. The fully connected layer can be used to perform a fully connected mapping operation on the third audio feature.

[0089] Figure 5 The model architecture diagram of the audio processing model of the present disclosure embodiment is schematically shown. Referring to Figure 5 , the audio processing model 5 can include a convolution unit 51, a time domain feature splitting processing unit 52, a time domain feature splitting processing unit 53, a time domain feature splitting processing unit 54, a convolution unit 55, an attention layer 56, and a fully connected layer 57.

[0090] The convolution unit 51 and the convolution unit 55 can both perform the above convolution processing, that is, they are a combination of one-dimensional convolution layer, linear rectification layer, and batch normalization layer operation. For the convolution unit 55, the outputs of the time domain feature splitting processing units are accumulated as the input of the convolution unit 55, thereby realizing the fusion of multiple layers of features and helping to improve the accuracy of speech detection.

[0091] The attention layer 56 can be configured in the form of a combination of attention (Attention) and batch normalization (BN), and the attention mechanism is directly applied to the time sequence dimension, so that the model can better learn the difference between speech and non-speech in long time sequence.

[0092] The fully connected layer 57 can map the output of the previous layer to the output node. The output node corresponding to the speech detection of the present disclosure has two, respectively representing the two states of speech and non-speech.

[0093] The time domain feature splitting processing units 52, 53, and 54 can be included in the time domain feature splitting processing module. However, it should be noted that the number of time domain feature splitting processing units included in the time domain feature splitting processing module can be one or more. It should be noted that each time domain feature splitting processing unit corresponds to performing the above time domain feature splitting sub-process.

[0094] The following refers to Figure 6 The time domain feature splitting processing unit of the present disclosure is described.

[0095] Referring to Figure 6 , the first convolution unit is used to transform the feature channels of the audio features input into the time domain feature splitting processing unit, and the output of this layer is x. It can be understood that x is an audio feature sequence. Then, x is split into 4 sub-sequences on the time sequence, denoted as x1, x2, x3 and x4.

[0096] Next, x1 remains unchanged, so y1 = x1. One-dimensional convolution processing of the convolution unit is performed on x2, x3 and x4 respectively, wherein the calculation result of the previous sub-sequence on the time sequence is added to the current sub-sequence, and then convolution processing is performed, thereby generating y2, y3 and y4. Then, y1, y2, y3 and y4 are spliced to generate y.

[0097] One-dimensional convolution processing of the convolution unit is performed on y to transform the feature dimension of the audio features to the feature dimension before inputting into the time domain feature splitting processing unit.

[0098] Then, the features are input into the compression excitation unit to achieve data enhancement. The compression excitation unit can include a compression network and an excitation network.

[0099] For example, the dimension of the audio features input into the unit is denoted as T*C, wherein T represents the time dimension and C represents the number of channels of the features. In this case, the compression network can compress the T*C audio features into 1*C audio features, which can be achieved by averaging the time dimension. After the T*C audio features are compressed into 1*C audio features, the one-dimensional parameters have a wider field of view and a wider receptive field than before. The excitation model can include a fully connected layer for predicting the importance of each channel of the 1*C audio features to obtain importance prediction results of different channels, and then applying the importance prediction results to the corresponding channels, thereby enhancing the features useful for audio processing tasks and suppressing the features useless or less useful for audio processing tasks.

[0100] As Figure 6 shown, the audio features output by the compression excitation unit can be added to the audio features input into the time domain feature splitting processing unit to form the final output of the time domain feature splitting processing unit. This connection method can be called skip connection, which can solve the problem of network degradation and also make the model converge quickly.

[0101] For the audio processing model of the embodiment of the present disclosure, in order to improve its robustness, the present disclosure further provides a scheme for data preprocessing of training data.

[0102] Specifically, when training the audio processing model, the original training data can be obtained in advance, and the original training data can be subjected to the operation of adding noise and / or reverberation to obtain intermediate training data. Subsequently, the intermediate training data can be subjected to an initial feature extraction operation to obtain first training audio features. Then, the audio processing model can be trained using the first training audio features.

[0103] In some embodiments of the present disclosure, the first training audio features can be directly used as data for training the audio processing model.

[0104] In some other embodiments of the present disclosure, the first training audio features can also be subjected to spectral enhancement processing to obtain second training audio features, and the audio processing model can be trained using the second training audio features.

[0105] Figure 7 A schematic diagram of a generation process of training data for training an audio processing model according to an embodiment of the present disclosure is shown. It should be understood that the present disclosure does not limit the order of the operation steps contained in the data preprocessing. Figure 7 The order of the operation steps contained in the data preprocessing is not limited, for example, reverberation can be added first and then noise can be added.

[0106] For adding noise, an open-source MUSAN dataset can be used, which contains three types of noise, i.e., speech, music, and noise. The specific method for adding noise is as follows: a type of noise is randomly selected for each training data to add noise. For speech noise, the signal-to-noise ratio of the added noise is 0 dB-15 dB, which can be randomly selected; for music noise, the signal-to-noise ratio of the added noise is 5 dB-15 dB, which can be randomly selected; for noise noise, the signal-to-noise ratio of the added noise is 10 dB-20 dB, which can be randomly selected. Two or more types of noise can also be added at the same time. This dataset covers a variety of noise types, and the noise adding process covers different signal-to-noise ratios, so it can simulate complex real-world scenarios to some extent and improve the accuracy of the model.

[0107] For adding reverberation, an open-source RIRS dataset can be used. The specific method is as follows: a piece of reverberation data is randomly selected to add reverberation to the training data. This dataset covers a large number of reverberation data, and the added reverberation can better simulate the reverberation scene of real users and improve the accuracy of the model.

[0108] For extracting features, the extraction method of the Fbank feature described above can be used.

[0109] For spectral enhancement, 0-5 frames can be randomly masked in the time dimension, and 0-10 signals can be randomly masked in the frequency domain. The masking can refer to setting all the data to 0, for example. Through spectral enhancement, the robustness of the model can be effectively improved.

[0110] Reference Figure 7 The original training data is generated into target training data after data preprocessing, and the target training data can be used alone or in combination with the original training data as a training sample to train the audio processing model of the embodiments of the present disclosure.

[0111] Based on the above audio processing process, the speech detection result of the audio data can be obtained, and then the subsequent algorithm can be executed using the speech detection result.

[0112] Figure 8 The flowchart of the video processing method according to the exemplary embodiments of the present disclosure is schematically shown. Referring to Figure 8 The video processing method can include the following steps:

[0113] S82. Obtain a video and extract audio data in the video.

[0114] In the exemplary embodiments of the present disclosure, the video can come from the photo album of the terminal device, and the terminal device can extract the audio data in the video.

[0115] S84. Perform speech detection on the audio data using the audio processing method to determine the speech segment in the video.

[0116] After the audio data is determined, the terminal device can perform speech detection on the audio data using the above-mentioned audio processing method of the present disclosure to determine the speech segment contained in the video.

[0117] Next, the terminal device can also analyze the speech segment, and separate the speech data of the target voice from the speech segment in the case that the speech segment contains multiple voices. The target voice can be a voice specified by the user.

[0118] Subsequently, the terminal device can generate a speech segment and / or a video segment corresponding to the target voice using the speech data of the target voice.

[0119] Specifically, in the embodiment of generating the speech segment corresponding to the target voice, the scattered speech data of the target voice can be spliced to generate the speech segment corresponding to the target voice.

[0120] In the embodiment of generating the video clip corresponding to the target human voice, the image frame aligned with the voice data of the target human voice can be determined from the video, and the voice data of the target human voice and the determined image frame are spliced respectively, and after time alignment, the video clip corresponding to the target human voice is generated.

[0121] In addition, for the video clip, the displayed image frame can also be irrelevant to the original video. That is, the voice of the target human voice can be combined with other image frames to generate an original video clip containing the target human voice.

[0122] Figure 9 The flowchart of constructing the wonderful child voice memory is schematically shown.

[0123] In step S902, the terminal device can obtain the video of the user album.

[0124] In step S904, the terminal device can perform activity voice detection by using the audio processing scheme of the embodiment of the present disclosure.

[0125] In step S906, the terminal device can perform human voice segmentation and clustering to divide different human voices in the video.

[0126] In step S908, the terminal device can detect each human voice data to determine the child voice.

[0127] In step S910, the terminal device can construct the wonderful child voice memory based on the voice data of the child voice, which can be in the form of sound or sound plus image, which is not limited by the present disclosure.

[0128] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps, etc.

[0129] Further, the present example embodiment also provides an audio processing device.

[0130] Figure 10 The block diagram of the audio processing device of the exemplary embodiment of the present disclosure is schematically shown. Referring to Figure 10 The audio processing device 10 according to the exemplary embodiment of the present disclosure can include a first feature determination module 101, a second feature determination module 103, a third feature determination module 105, and a voice detection module 107.

[0131] Specifically, the first feature determination module 101 can be configured to perform an initial feature extraction operation on the audio data to obtain first audio features; the second feature determination module 103 can be configured to perform a time-domain feature splitting based processing procedure on the first audio features to obtain second audio features; the third feature determination module 105 can be configured to enhance the difference between speech features and non-speech features in the second audio features based on an attention mechanism to obtain third audio features; and the speech detection module 107 can be configured to perform a fully connected mapping operation on the third audio features to obtain a speech detection result of the audio data.

[0132] According to an example embodiment of the present disclosure, the time-domain feature splitting based processing procedure includes at least one level of time-domain feature splitting sub-processes. In this case, the second feature determination module 103 can be configured to perform: performing at least one level of time-domain feature splitting sub-processes on the first audio features; and generating the second audio features based on the output results of the at least one level of time-domain feature splitting sub-processes.

[0133] According to an example embodiment of the present disclosure, in the case of performing more than two levels of time-domain feature splitting sub-processes on the first audio features, the output of each level of time-domain feature splitting sub-processes is the input of the next level of time-domain feature splitting sub-processes. In this case, the process of the second feature determination module 103 for generating the second audio features based on the output results of the more than two levels of time-domain feature splitting sub-processes can be configured to perform: fusing the output results of each level of time-domain feature splitting sub-processes; and determining the second audio features according to the fused results.

[0134] According to an example embodiment of the present disclosure, the time-domain feature splitting sub-process performed by the second feature determination module 103 includes: performing a time-domain feature splitting operation on the audio features input to the time-domain feature splitting sub-process to obtain a plurality of first intermediate features; performing convolution processing on the plurality of first intermediate features to obtain a plurality of second intermediate features; splicing the plurality of second intermediate features to obtain a third intermediate feature; and performing feature enhancement on the third intermediate feature to obtain the audio features output by the time-domain feature splitting sub-process.

[0135] According to an example embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: performing convolution processing on the audio features input to the time-domain feature splitting sub-process, and performing a time-domain feature splitting operation on the features obtained after the convolution processing to obtain a plurality of first intermediate features.

[0136] According to an example embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: taking the first first intermediate feature in the time domain as a first second intermediate feature; performing convolution processing on the second first intermediate feature in the time domain to obtain a second second intermediate feature; adding the i-th first intermediate feature in the time domain and the (i-1)-th second intermediate feature, and performing convolution processing on the added feature to obtain the i-th second intermediate feature; wherein i is a positive integer greater than or equal to 3.

[0137] According to an example embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: performing feature enhancement on the third intermediate feature to obtain a fourth intermediate feature; adding the fourth intermediate feature and the audio feature of the input time domain feature splitting sub-process to obtain the audio feature output by the time domain feature splitting sub-process.

[0138] According to an example embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: performing convolution processing on the third intermediate feature, and performing feature enhancement on the feature obtained after the convolution processing to obtain a fourth intermediate feature.

[0139] According to an example embodiment of the present disclosure, the second feature determination module 103 can be configured to perform: determining the importance of each feature channel in the third intermediate feature; and configuring a weight for each feature channel according to the importance of each feature channel.

[0140] According to an example embodiment of the present disclosure, the speech detection module 107 can be configured to perform: performing a fully connected mapping operation on the third audio feature to determine a speech detection result of each audio frame in the audio data; wherein the speech detection result represents a detection result of whether the audio frame is speech or non-speech.

[0141] According to an example embodiment of the present disclosure, the speech detection module 107 can be configured to perform: performing a fully connected mapping operation on the third audio feature to determine a first speech detection result of each audio frame in the audio data; and correcting the first speech detection result by using a first frame number threshold and a second frame number threshold to determine a second speech detection result of each audio frame in the audio data; wherein the first frame number threshold is a frame number requirement set by a speech state transition to a non-speech state, and the second frame number threshold is a frame number requirement set by a non-speech state transition to a speech state.

[0142] According to an example embodiment of the present disclosure, the first feature determination module 101 can be configured to perform: performing an initial feature extraction operation on the audio data to obtain an initial feature; and performing convolution processing on the initial feature to obtain a first audio feature.

[0143] According to the example embodiments of the present disclosure, the audio processing model can be trained to implement the above processing procedure. In this case, the audio processing device 11 can further include a model training module 111, compared with the audio processing device 10. Figure 11

[0144] Specifically, the model training module 111 can be configured to perform: pre-acquiring original training data; performing noise adding and / or reverberation operation on the original training data to obtain intermediate training data; performing initial feature extraction operation on the intermediate training data to obtain first training audio features; and training the audio processing model by using the first training audio features.

[0145] According to the example embodiments of the present disclosure, the model training module 111 can be further configured to perform: performing spectral enhancement processing on the first training audio features to obtain second training audio features; and training the audio processing model by using the second training audio features.

[0146] Further, the example embodiments of the present disclosure also provide a video processing device.

[0147] Figure 12 A block diagram of an audio processing device according to the example embodiments of the present disclosure is shown schematically. Referring to Figure 12 According to the example embodiments of the present disclosure, the audio processing device 12 can include an audio extraction module 121 and a segment determination module 123.

[0148] Specifically, the audio extraction module 121 can be configured to acquire a video and extract audio data in the video; and the segment determination module 123 can be configured to perform voice detection on the audio data by using the above audio processing method to determine a voice segment in the video.

[0149] According to the example embodiments of the present disclosure, the segment determination module 123 can be further configured to perform: in a case where the voice segment contains multiple human voices, separating voice data of a target human voice from the voice segment; and generating a voice segment and / or a video segment corresponding to the target human voice by using the voice data of the target human voice.

[0150] Since the various functional modules of the audio processing device and the video processing device according to the example embodiments of the present disclosure are the same as those in the above method embodiments, no further description is given herein.

[0151] Figure 13 A schematic diagram of an electronic device suitable for implementing the example embodiments of the present disclosure is shown. The terminal device according to the example embodiments of the present disclosure can be configured in the form as shown in Figure 13 It should be noted that, Figure 13 The electronic device shown is only an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.​

[0152] The electronic device disclosed herein includes at least a processor and a memory, the memory being used to store one or more programs, which, when executed by the processor, enable the processor to implement the audio processing method and / or video processing method of the exemplary embodiments of this disclosure.

[0153] Specifically, such as Figure 13 As shown, the electronic device 130 may include: a processor 1310, internal memory 1321, external memory interface 1322, Universal Serial Bus (USB) interface 1330, charging management module 1340, power management module 1341, battery 1342, antenna 1, antenna 2, mobile communication module 1350, wireless communication module 1360, audio module 1370, sensor module 1380, display screen 1390, camera module 1391, indicator 1392, motor 1393, buttons 1394, and a Subscriber Identification Module (SIM) card interface 1395, etc. The sensor module 1380 may include a depth sensor, pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, proximity sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, and bone conduction sensor, etc.

[0154] It is understood that the structures illustrated in the embodiments of this disclosure do not constitute a specific limitation on the electronic device 130. In other embodiments of this disclosure, the electronic device 130 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0155] Processor 1310 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. Additionally, processor 1310 may include memory for storing instructions and data.

[0156] The electronic device 130 can implement the photographing function through an ISP, a camera module 1391, a video codec, a GPU, a display screen 1390, and an application processor, and the like. In some embodiments, the electronic device 130 can include 1 or N camera modules 1391, N being a positive integer greater than 1, and if the electronic device 130 includes N cameras, one of the N cameras is a main camera.

[0157] The internal memory 1321 can be used to store computer executable program codes including instructions. The internal memory 1321 can include a program storage area and a data storage area. The external memory interface 1322 can be used to connect with an external memory card, such as a Micro SD card, to expand the memory capacity of the electronic device 130.

[0158] The present disclosure also provides a computer readable storage medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device.

[0159] The computer readable storage medium may, for example, be— but is not limited to— an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program used or used in conjunction with an instruction execution system, apparatus, or device.

[0160] The computer readable storage medium can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0161] The computer readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method as described in the embodiments of the present disclosure.

[0162] The flow and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0163] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware, or by a combination of software and hardware. The names of the units described are not intended to constitute a limitation on the units per se.

[0164] From the above description of the embodiments, it can be readily appreciated by those skilled in the art that the example embodiments described herein can be realized by software and also realized by a combination of software and necessary hardware. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a nonvolatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, or the like) or a network, and includes a number of instructions for causing one computer (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the methods according to the embodiments of the present disclosure.

[0165] In addition, the above-described flowcharts are only schematic representations of processes comprising the present method, and the order of the processes is not limited to that shown in the flowcharts, unless otherwise specified. It is readily apparent to a person skilled in the art that the processes of the method can be performed in a different order, or that some of the processes can be performed in parallel, unless otherwise specified.

[0166] It should be noted that although several modules or units are mentioned in the foregoing detailed description for the device performing actions, the division into such modules or units is not mandatory. In fact, according to the embodiments of the present disclosure, features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functions of one module or unit described above can be further split into several modules or units.

[0167] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the concepts disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0168] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. An audio processing method, characterized in that, include: Initial feature extraction is performed on the audio data to obtain the first audio features; The first audio feature is processed by temporal feature splitting to obtain the second audio feature; The third audio feature is obtained by enhancing the difference between speech features and non-speech features in the second audio feature based on the attention mechanism. A fully connected mapping operation is performed on the third audio feature to determine the speech detection result for each audio frame in the audio data; the speech detection result represents whether the audio frame is speech or non-speech. Specifically, performing a fully connected mapping operation on the third audio feature to determine the speech detection result for each audio frame in the audio data includes: A fully connected mapping operation is performed on the third audio feature to determine the first speech detection result for each audio frame in the audio data; The first speech detection result is corrected using a first frame count threshold and a second frame count threshold to determine the second speech detection result for each audio frame in the audio data; Wherein, the first frame count threshold is the frame count requirement set when switching from voice state to non-voice state, and the second frame count threshold is the frame count requirement set when switching from non-voice state to voice state.

2. The audio processing method according to claim 1, characterized in that, The time-domain feature segmentation process includes at least one time-domain feature segmentation sub-process; wherein, performing the time-domain feature segmentation process on the first audio feature to obtain the second audio feature includes: Perform at least one level of temporal feature decomposition subprocessing on the first audio feature; The second audio feature is generated based on the output of the at least one-level time-domain feature decomposition subprocess.

3. The audio processing method according to claim 2, characterized in that, When performing two or more levels of temporal feature decomposition subprocessing on the first audio feature, the output of each level of temporal feature decomposition subprocessing is the input of the next level of temporal feature decomposition subprocessing; wherein, generating the second audio feature based on the output of the two or more levels of temporal feature decomposition subprocessing includes: The outputs of each level of time-domain feature decomposition subprocessing are fused; The second audio feature is determined based on the fusion result.

4. The audio processing method according to claim 2, characterized in that, The time-domain feature decomposition process includes: The audio features input to the time-domain feature decomposition subprocess are subjected to time-domain feature decomposition to obtain multiple first intermediate features; The multiple first intermediate features are convolved to obtain multiple second intermediate features; The multiple second intermediate features are concatenated to obtain the third intermediate feature; The third intermediate feature is enhanced to obtain the audio features output by the temporal feature decomposition subprocess.

5. The audio processing method according to claim 4, characterized in that, The audio features input to the time-domain feature decomposition subprocess are subjected to time-domain feature decomposition to obtain multiple first intermediate features, including: The audio features input to the temporal feature decomposition subprocess are subjected to convolution processing, and the features obtained after convolution processing are subjected to temporal feature decomposition operation to obtain multiple first intermediate features.

6. The audio processing method according to claim 4, characterized in that, The plurality of first intermediate features are convolved to obtain a plurality of second intermediate features, including: The first intermediate feature in the time domain is used as the first second intermediate feature; The second first intermediate feature in the time domain is convolved to obtain the second second intermediate feature; Add the i-th first intermediate feature and the (i-1)-th second intermediate feature in the time domain, and then perform convolution on the added feature to obtain the i-th second intermediate feature; Where i is a positive integer greater than or equal to 3.

7. The audio processing method according to claim 4, characterized in that, Feature enhancement is performed on the third intermediate feature to obtain the audio features output by the temporal feature decomposition subprocess, including: The third intermediate feature is enhanced to obtain the fourth intermediate feature; The fourth intermediate feature is added to the audio feature input to the time-domain feature splitting subprocess to obtain the audio feature output by the time-domain feature splitting subprocess.

8. The audio processing method according to claim 7, characterized in that, The third intermediate feature is enhanced to obtain a fourth intermediate feature, including: The third intermediate feature is convolved, and the feature obtained after convolution is enhanced to obtain the fourth intermediate feature.

9. The audio processing method according to claim 4, characterized in that, Feature enhancement of the third intermediate feature includes: Determine the importance of each feature channel in the third intermediate feature; Each feature channel is assigned a weight based on its importance.

10. The audio processing method according to claim 1, characterized in that, Initial feature extraction is performed on the audio data to obtain the first audio features, including: Initial feature extraction is performed on the audio data to obtain initial features; The initial features are convolved to obtain the first audio features.

11. The audio processing method according to any one of claims 1 to 10, characterized in that, The audio processing method is implemented using an audio processing model; The audio processing model includes: The temporal feature splitting processing module is used to perform a temporal feature splitting process on the first audio feature to obtain the second audio feature; An attention layer is used to enhance the difference between speech features and non-speech features in the second audio feature based on an attention mechanism, so as to obtain the third audio feature; A fully connected layer is used to perform a fully connected mapping operation on the third audio feature.

12. The audio processing method according to claim 11, characterized in that, The audio processing method further includes: Acquire raw training data in advance; The original training data is subjected to noise and / or reverberation to obtain intermediate training data; An initial feature extraction operation is performed on the intermediate training data to obtain the first training audio features; The audio processing model is trained using the first training audio features.

13. The audio processing method according to claim 12, characterized in that, Training the audio processing model using the first training audio features includes: The first training audio features are subjected to spectral enhancement processing to obtain the second training audio features; The audio processing model is trained using the second training audio features.

14. A video processing method, characterized in that, include: Acquire the video and extract the audio data from the video; The audio data is subjected to speech detection using the audio processing method according to any one of claims 1 to 13 to determine speech segments in the video.

15. The video processing method according to claim 14, characterized in that, The video processing method further includes: When the speech segment contains multiple human voices, the speech data of the target human voice is extracted from the speech segment; Use the speech data of the target human voice to generate speech segments and / or video segments corresponding to the target human voice.

16. An audio processing apparatus, characterized in that, include: The first feature determination module is used to perform initial feature extraction on the audio data to obtain the first audio feature; The second feature determination module is used to perform a time-domain feature splitting process on the first audio feature to obtain the second audio feature; The third feature determination module is used to enhance the difference between speech features and non-speech features in the second audio feature based on an attention mechanism, so as to obtain the third audio feature; The speech detection module is used to perform a fully connected mapping operation on the third audio feature to determine the speech detection result of each audio frame in the audio data; the speech detection result represents whether the audio frame is speech or non-speech. Specifically, performing a fully connected mapping operation on the third audio feature to determine the speech detection result for each audio frame in the audio data includes: A fully connected mapping operation is performed on the third audio feature to determine the first speech detection result for each audio frame in the audio data; The first speech detection result is corrected using a first frame count threshold and a second frame count threshold to determine the second speech detection result for each audio frame in the audio data; Wherein, the first frame count threshold is the frame count requirement set when switching from voice state to non-voice state, and the second frame count threshold is the frame count requirement set when switching from non-voice state to voice state.

17. A video processing apparatus, characterized in that, include: An audio extraction module is used to acquire a video and extract audio data from the video. A segment determination module is used to perform speech detection on the audio data using the audio processing method according to any one of claims 1 to 13, so as to determine the speech segments in the video.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the audio processing method as described in any one of claims 1 to 13 or the video processing method as described in claim 14 or 15.

19. An electronic device, characterized in that, include: processor; A memory for storing one or more programs that, when executed by the processor, cause the processor to implement the audio processing method as described in any one of claims 1 to 13 or the video processing method as described in claim 14 or 15.

Images