Feature processing method, device and device for processing features

Abstract

The embodiments of the present application disclose a feature processing method, device and device for processing features. The embodiments of the method include: performing frame and window processing on audio data, and extracting audio frame features of each window; inputting the audio frame features of the first window into a convolutional neural network to obtain the feature processing results of the first window; taking each of the remaining windows as a target window in turn, and performing the following steps: determining the repeated frame features and non-repeated frame features of the target window relative to the previous window; determining the feature processing results of the non-repeated frame features based on the convolutional neural network, extracting the feature processing results of the repeated frame features from the feature processing results of the previous window, and determining the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features. This implementation avoids the waste of computing resources and improves the processing efficiency of audio data.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a feature processing method, device, and a device for processing features. Background Art

[0002] Sequence modeling refers to a method of converting a feature sequence of an audio segment into a fixed-length vector representation. Usually, a convolutional neural network is used to convert the features and obtain the feature processing results in vector form.

[0003] In the prior art, usually after the audio data is framed and windowed, the audio frame features of each window are directly input into the convolutional neural network to obtain the feature processing results. Since there are repeated frames between adjacent windows, there is a problem of repeated calculation, which causes a waste of computing resources and slows down the processing efficiency of audio data. Summary of the invention

[0004] The embodiments of the present application propose a feature processing method, a device, and a device for processing features to solve the technical problems of waste of computing resources and low processing efficiency of audio data in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a feature processing method, the method comprising: performing frame and window processing on audio data, and extracting audio frame features of each window; inputting the audio frame features of the first window into a convolutional neural network to obtain a feature processing result of the first window; taking each of the remaining windows as a target window in turn, and performing the following steps: determining the repeated frame features and non-repeated frame features of the target window relative to the previous window; determining the feature processing results of the non-repeated frame features based on the convolutional neural network, extracting the feature processing results of the repeated frame features from the feature processing results of the previous window, and determining the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features.

[0006] In a second aspect, an embodiment of the present application provides a feature processing device, which includes: a windowing unit, used to perform frame and window processing on audio data, and extract audio frame features of each window; a first processing unit, used to input the audio frame features of the first window into a convolutional neural network to obtain a feature processing result of the first window; a second processing unit, used to take each of the remaining windows as a target window in turn, and perform the following steps: determine the repeated frame features and non-repeated frame features of the target window relative to the previous window; determine the feature processing results of the non-repeated frame features based on the convolutional neural network, extract the feature processing results of the repeated frame features from the feature processing results of the previous window, and determine the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features.

[0007] In a third aspect, an embodiment of the present application provides a device for processing features, comprising a memory, and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs include instructions for performing the following operations: performing frame and window processing on audio data, and extracting audio frame features of each window; inputting the audio frame features of the first window into a convolutional neural network to obtain a feature processing result of the first window; taking each of the remaining windows as a target window in turn, determining the repeated frame features and non-repeated frame features of the target window relative to the previous window, extracting the feature processing results of the repeated frame features from the feature processing results of the previous window, determining the feature processing results of the non-repeated frame features based on the convolutional neural network, and determining the feature processing results of the target window based on the feature processing results of the repeated frame features and the feature processing results of the non-repeated frame features.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0009] The feature processing method, device and device for processing features provided by the embodiment of the present application perform frame and window processing on the audio data and extract the audio frame features of each window; then input the audio frame features of the first window into the convolutional neural network to obtain the feature processing result of the first window; finally, take each of the remaining windows as the target window in turn, determine the repeated frame features and non-repeated frame features of the target window relative to the previous window, determine the feature processing results of the non-repeated frame features based on the convolutional neural network, extract the feature processing results of the repeated frame features from the feature processing results of the previous window, and determine the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features. Thus, when performing feature processing on each window, the processing results of the repeated frame features of the previous window can be directly copied, and only the non-repeated frame features are processed, thereby avoiding repeated calculations of the same audio frame features, thereby avoiding waste of computing resources and improving the processing efficiency of audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is a flow chart of an embodiment of a feature processing method according to the present application;

[0012] Figure 2 It is a schematic diagram of the feature processing process of two adjacent windows by a convolutional neural network in the feature processing method of the present application;

[0013] Figure 3 is a structural schematic diagram of an embodiment of a feature processing device according to the present application;

[0014] Figure 4 is a schematic structural diagram of a device for processing features according to the present application;

[0015] Figure 5 It is a schematic diagram of the structure of the server in some embodiments of the present application. DETAILED DESCRIPTION

[0016] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] Please refer to Figure 1 , which shows a process 100 according to an embodiment of the feature processing method of the present application. The above feature processing method can be run on various electronic devices, including but not limited to: servers, smart phones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, car computers, desktop computers, set-top boxes, smart TVs, wearable devices, etc.

[0019] The feature processing method in this embodiment may include the following steps:

[0020] Step 101, perform frame division and windowing processing on the audio data, and extract the audio frame features of each window.

[0021] In this embodiment, the execution subject of the feature processing method (such as the above-mentioned electronic device) can first perform frame windowing processing on the audio data to be processed. Among them, the frame windowing processing may include frame processing and windowing processing. Frame processing refers to segmenting the audio data according to a specified length (time period or number of samples) to obtain short-term stable audio frames. Windowing processing refers to multiplying the speech data with a window function (such as a Hamming window function) to obtain a window (also called a sliding window) containing several audio frames.

[0022] In this embodiment, after the audio data is framed and windowed, the execution subject can extract the audio frame features of each window. The audio frame features are the features of the audio frame. The audio frame features may include but are not limited to low energy frame rate, maximum bandwidth, and pitch period credibility standard deviation. Since each window may contain several audio frames, the audio frame features of each window may be a feature sequence.

[0023] Step 102: Input the audio frame features of the first window into the convolutional neural network to obtain the feature processing result of the first window.

[0024] In this embodiment, the execution subject may pre-store a convolutional neural network for feature processing, such as a convolutional neural network TCN (Temporal Convolutional Network) for sequence modeling. The execution subject may input the audio frame features of the first window into the convolutional neural network to obtain the feature processing result of the first window. The network structure of the convolutional neural network is not specifically limited here.

[0025] In some optional implementations of this embodiment, the convolutional neural network may include at least one convolutional layer. The feature processing result of the first window may include the convolution calculation result of each convolutional layer in the convolutional neural network for the first window. After the audio frame feature of the first window is input into the convolutional neural network, each convolutional layer in the convolutional neural network may perform feature processing in turn, and finally output the convolution calculation result of the last convolutional layer.

[0026] Specifically, the convolutional layers in the convolutional neural network can be connected in sequence, and the output of the previous convolutional layer can be used as the input of the next convolutional layer. The above-mentioned execution entity can first input the audio frame features of the first window into the first convolutional layer of the convolutional neural network to obtain the convolution calculation result of the first convolution layer for the first window. Then, for each convolution layer starting from the second convolutional layer, the convolution calculation result of the previous convolutional layer for the first window is input into the convolutional layer to obtain the convolution calculation result of the convolution layer for the first window. In this way, the convolution calculation results of each convolutional layer for the first window can be obtained, and the convolution calculation result of the last convolutional layer can be output.

[0027] It should be noted that, for each convolution layer starting from the second convolution layer, before the convolution calculation result of the previous convolution layer for the first window is input to the convolution layer, the convolution calculation result of the previous convolution layer may be frame-padded based on the frame-padded parameters of the convolution layer, so that the data length input to the convolution layer meets the requirements. The frame-padded parameters may include left frame-padded parameters and right frame-padded parameters.

[0028] The following is an explanation using a specific example. The window length is W (i.e., the number of audio frames contained in each window, i.e., the number of audio frame features corresponding to each window), and the window moving step length between adjacent sliding windows is H. The convolutional neural network contains N convolutional layers. The number of input features of each convolutional layer is W[1], W[2], …, W[N], respectively. The left-padded frame parameters of each convolutional layer are L[1], L[2], …, L[N], respectively, and the right-padded frame parameters are R[1], R[2], …, R[N], respectively. The convolution kernel size of each convolutional layer is K[1], K[2], …, K[N], respectively. The stride parameter (Stride), dilation parameter (Dilation) and grouping parameter (Groups) of each convolutional layer in the convolutional neural network are all 1.

[0029] The above execution subject can first perform feature processing on the first window, that is, input the W frame features corresponding to the first window as a whole into the convolutional neural network. The specific processing process is as follows:

[0030] In the first step, the first convolutional layer (i.e., the first convolutional layer) of the convolutional neural network is used to perform feature processing on the first window. At this time, W[1] = W. First, according to the left and right complement parameters of the first convolutional layer, the L[1] frame features are supplemented on the left side of W[1], and the R[1] frame features are supplemented on the right side of W[1], and a total of (L[1] + W[1] + R[1]) frame features are obtained. After that, the (L[1] + W[1] + R[1]) frame features are input to the first convolutional layer to obtain (L[1] + W[1] + R[1])-(K[1]-1)) = W[2] frame output.

[0031] In the second step, the second convolutional layer of the convolutional neural network is used to process the features of the first window. At this time, W[2] = ((L[1] + W[1] + R[1]) - (K[1] - 1)). First, according to the left and right complement parameters of the second convolutional layer, the L[2] frame is supplemented on the left side of W[2], and the R[2] frame is supplemented on the right side of W[2], and a total of (L[2] + W[2] + R[2]) frame features are obtained. After that, the (L[2] + W[2] + R[2]) frame is input to the second convolutional layer, and the ((L[2] + W[2] + R[2]) frame output is obtained.

[0032] By analogy, the above execution entity can repeat this process for each subsequent convolutional layer until all N convolutional layers are calculated and the ((L[N]+W[N]+R[N])-(K[N]-1)) frame output is obtained.

[0033] Step 103, taking each of the remaining windows as a target window in turn, and executing the following steps: determining the repeated frame features and non-repeated frame features of the target window relative to the previous window; determining the feature processing results of the non-repeated frame features based on a convolutional neural network, extracting the feature processing results of the repeated frame features from the feature processing results of the previous window, and determining the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features.

[0034] In this embodiment, the execution subject can sequentially use each window from the second window as the target window, and first determine the repeated frame features and non-repeated frame features of the target window relative to the previous window. For example, if the window length is W, and the window shift between two adjacent frames (i.e., the window shift step is H), then there is an overlap of (WH) frames between the two adjacent windows. This (WH) frame is a repeated frame, and the feature of this (WH) frame is a repeated frame feature. The feature processing results of the repeated frame features are the same. Features other than the repeated frame features are non-repeated frame features.

[0035] The execution subject may determine the feature processing result of the non-repeated frame feature based on the convolutional neural network, such as inputting the non-repeated frame feature into the convolutional neural network to obtain the feature processing result of the non-repeated frame feature. At the same time, since the feature processing results of the repeated frames of two adjacent windows are the same, the execution subject may extract the feature processing result of the repeated frame feature from the feature processing result of the previous window, and obtain the feature processing result of the target window based on the feature processing result of the non-repeated frame feature and the feature processing result of the repeated frame feature (such as summarizing the two).

[0036] Since the processing results of the repeated frame features of the previous window can be directly copied when performing feature processing on each window, only the non-repeated frame features are processed, thereby avoiding repeated calculation of the same audio frame features, avoiding waste of computing resources, and improving the processing efficiency of audio data.

[0037] In some optional implementations of this embodiment, the convolutional neural network may include at least one convolutional layer. The feature processing result of each window may include the convolution calculation result of each convolutional layer in the convolutional neural network for the window. Here, the convolutional layers in the convolutional neural network may be connected in sequence, and the output of the previous convolutional layer may be used as the input of the next convolutional layer. For a certain window, the input features of the first convolutional layer may include the audio frame features of the window. The input features of the remaining convolutional layers may include the convolution calculation result for the window output by the previous convolutional layer.

[0038] In some optional implementations of the present embodiment, the above-mentioned execution entity may divide the input features of each window in each convolutional layer into a first part, a second part, and a third part based on a preset window length (such as W), a preset window sliding step size (such as H), and a convolution kernel size in each convolutional layer in a convolutional neural network (such as K[1], K[2], ..., K[N] in sequence). The second part may correspond to repeated frames, and the first part and the third part may correspond to non-repeated frames. Afterwards, the above-mentioned execution entity may use the second part of the input features of the target window in each convolutional layer as the repeated frame features of the target window relative to the previous window; and use the first part and the third part of the input features of the target window in each convolutional layer as the non-repeated frame features of the target window relative to the previous window.

[0039] Furthermore, for each convolutional layer of the convolutional neural network, the execution subject can input the first part and the third part of the input features of the target window in the convolutional layer into the convolutional layer, respectively, to obtain the first convolution calculation result and the third convolution calculation result, respectively; then, the convolution calculation result corresponding to the second part of the input features of the previous window in the convolutional layer can be extracted from the convolution calculation result of the convolutional layer for the previous window as the second convolution calculation result; finally, the first convolution calculation result, the second convolution calculation result and the third convolution calculation result can be summarized to obtain the convolution calculation result of the convolutional layer for the target window. Thus, when each convolutional layer of the convolutional neural network performs feature processing for each window, the processing result of the repeated frame features of the previous window can be directly copied by the convolutional layer, thereby avoiding repeated calculation of the same audio frame features of adjacent windows by the same convolutional layer, thereby avoiding waste of computing resources and improving the processing efficiency of audio data.

[0040] In the above implementation, before using the convolution layer for calculation, the above execution body can also perform frame complement processing on the input features so that the number of features input to the convolution layer meets the requirements. Specifically, for each convolution layer of the convolutional neural network, the target window can be frame complemented on the left side of the first part of the input features of the convolution layer based on the first frame complement parameter of the convolution layer to obtain the first complement audio frame feature; then the first complement audio frame feature is input to the convolution layer to obtain the first convolution calculation result. Similarly, for each convolution layer of the convolutional neural network, the target window can be frame complemented on the right side of the third part of the input features of the convolution layer based on the second frame complement parameter of the convolution layer to obtain the second complement audio frame feature; then the second complement audio frame feature is input to the convolution layer to obtain the third convolution calculation result.

[0041] Continue with the example in step 102 for further explanation and see Figure 2 . Figure 2Schematic diagram of the feature processing process of two adjacent windows by the convolutional neural network. For the first window (i.e., the first window), the above execution entity can divide the input frame number of the first window in each convolution layer into three parts, which are recorded as W[i]_left, W[i]_overlap and W[i]_right. W[i]_left+W[i]_overlap+W[i]_right=W[i]. Wherein, i is a natural number greater than or equal to 1 and less than or equal to N. W[i] is the input feature number of the i-th convolutional layer. W[1]=W. When i is greater than or equal to 2, W[i]=((L[i-1]+W[i-1]+R[i-1])-(K[i-1]-1)). Here, W[1]_left=H, W[1]_overlap=WH, and W[1]_right=0. When i is greater than or equal to 2, W[i]_left, W[i]_overlap and W[i]_right can be calculated according to the following formulas: W[i]_left = W[i-1]_left + L[i-1], W[i]_overlap = W[i-1]_overlap - (K[i-1] - 1), W[i]_right = W[i-1]_right + R[i-1]. The feature corresponding to W[i]_overlap is the repeated frame feature.

[0042] When each convolution layer processes the first window, the convolution calculation results of the repeated frame features of the first window and the second window (i.e., the features corresponding to W[i]_overlap) can be stored. That is, the (W[1]_overlap-(K[1]-1)) frames starting from the (W[1]_left+L[1]+1) frame of the first convolution layer can be saved. Therefore, when the first convolution layer performs feature processing on the second window, the saved convolution calculation results can be directly copied, and only the non-repeated frame features are processed. Here, if (W[1]_overlap-(K[1]-1)) is less than or equal to 0, it does not need to be saved. Similarly, the calculation results of the (W[2]_overlap-(K[2]-1)) frames starting from the (W[2]_left+L[2]+1) frame of the second convolutional layer can be saved. If (W[2]_overlap-(K[2]-1)) is less than or equal to 0, it is not saved. By analogy, N groups of calculation results can be saved. The saved calculation results can be copied when each convolutional layer performs feature processing on the next window to avoid repeated calculations on repeated frame features, thereby avoiding waste of computing resources and improving the processing efficiency of audio data.

[0043] The second window is obtained by sliding the first window backward by H frames. That is, the first H frames of the first window are removed, and new H frames are added after the remaining frames to form the second sliding window (still W frames). For the second window, the specific processing process is as follows:

[0044] Step S11, using the first convolutional layer of the convolutional neural network to perform feature processing on the second window.

[0045] First, the input frame number W[1] (W[1]=W) of the second window in the first convolutional layer can be divided into three parts, which are recorded as W[1]_left_calculate, W[1]_copy and W[1]_right_calculate. W[1]_left_calculate=W[1]_left-H+(K[1]-1), W[1]_copy=W[1]_overlap-(K[1]-1), W[1]_right_calculate=W[1]_right+H+(K[1]-1).

[0046] Then, the first convolutional layer of the convolutional neural network is used to perform convolution calculation on the W[1]_left_calculate part. Here, according to the left complement frame parameter of the first convolutional layer, the L[1] frame can be added to the left of the W[1]_left_calculate frame to obtain a total of (L[1]+W[1]_left_calculate) frame features. This is used as the input of the first convolutional layer to obtain the (L[1]+W[1]_left_calculate)-(K[1]-1) frame output, which is used as the first convolution calculation result.

[0047] Afterwards, the feature processing results of the second part, a total of W[1]_copy frames, are copied as the second convolution calculation results of the first convolution layer (i.e., the calculation results of the first convolution layer for the ((L[1]+W[1]+R[1])-(K[1]-1)) frame of the first window, starting from the (W[1]_left+L[1]+1) frame to the (W[1]_overlap-(K[1]-1)) frame calculation results).

[0048] Then, the first convolutional layer of the convolutional neural network is used to perform convolution calculation on the W[1]_right_calculate part. Here, according to the right complement frame parameter of the first convolutional layer, the R[1] frame can be added to the right of the W[1]_right_calculate frame to obtain a total of (W[1]_right_calculate+R[1]) frame features. This is used as the input of the first convolutional layer to obtain the (W[1]_right_calculate+R[1])-(K[1]-1) frame output, which is used as the result of the third convolution calculation.

[0049] Finally, the first convolution calculation result, the second convolution calculation result, and the third convolution calculation result can be combined in sequence to obtain the convolution calculation result of the first convolution layer for the second window. The total number of frames of the convolution calculation result is (L[1]+W[1]+R[1])-(K[1]-1).

[0050] Furthermore, the (W[1]_overlap-(K[1]-1)) frame starting from the (W[1]_left+L[1]+1) frame in the calculation result of the ((L[1]+W[1]+R[1])-(K[1]-1)) frame can be saved so that the convolution layer can directly use it when performing feature processing on the next window. Here, if (W[1]_overlap-(K[1]-1)) is less than or equal to 0, it does not need to be saved.

[0051] Step S12, using the second convolutional layer of the convolutional neural network to perform feature processing on the second window.

[0052] First, the input frame number W[2] of the second window in the second convolutional layer (W[2] = (L[1] + W[1] + R[1]) - (K[1] - 1)) can be divided into three parts, denoted as W[2]_left_calculate, W[2]_copy and W[2]_right_calculate. W[2]_left_calculate = W[2]_left - H + (K[2] - 1), W[2]_copy = W[2]_overlap - (K[2] - 1), W[2]_right_calculate = W[2]_right + H + (K[2] - 1).

[0053] Then, the second convolutional layer of the convolutional neural network is used to perform convolution calculation on the W[2]_left_calculate part. Here, according to the left-padded frame parameter of the second convolutional layer, the L[2] frame can be padded to the left of the W[2]_left_calculate frame to obtain a total of (L[2]+W[2]_left_calculate) frame features. This is used as the input of the second convolutional layer to obtain the (L[2]+W[2]_left_calculate)-(K[2]-1) frame output, which is used as the first convolution calculation result.

[0054] Afterwards, the feature processing results of the second part, a total of W[2]_copy frames, are copied as the second convolution calculation results of the second convolution layer (i.e., the calculation results of the second convolution layer for the (L[2]+W[2]+R[2])-(K[2]-1) frames of the first window, starting from the (W[2]_left+L[2]+1) frame to the (W[2]_overlap-(K[2]-1)) frame calculation results).

[0055] Then, the convolutional calculation is performed on the W[2]_right_calculate part using the second convolutional layer of the convolutional neural network. Here, the R[2] frame can be added to the right of the W[2]_right_calculate frame according to the right complement frame parameter of the second convolutional layer, and a total of (W[2]_right_calculate+R[2]) frame features are obtained. This is used as the input of the second convolutional layer to obtain the ((W[2]_right_calculate+R[2])-(K[2]-1)) frame output, which is used as the result of the third convolution calculation.

[0056] Finally, the first convolution calculation result, the second convolution calculation result, and the third convolution calculation result can be combined in sequence to obtain the convolution calculation result of the second convolution layer for the second window. The total number of frames of the convolution calculation result is (L[2]+W[2]+R[2])-(K[2]-1).

[0057] Furthermore, the (W[2]_overlap-(K[2]-1)) frame starting from the (W[2]_left+L[2]+1) frame in the calculation result of the (L[2]+W[2]+R[2])-(K[2]-1) frame can be saved so that it can be directly used when the convolution layer performs feature processing on the next window. Here, if (W[2]_overlap-(K[2]-1)) is less than or equal to 0, it does not need to be saved.

[0058] Similarly, the remaining convolutional layers of the convolutional neural network can be calculated until the N convolutional layers of the convolutional neural network are calculated, and the (L[N]+W[N]+R[N])-(K[N]-1) frame calculation results are obtained. If (W[i]_overlap-(K[i]-1)) is greater than 0, the calculation is performed in the manner described in steps S11 and S12 above. If (W[i]_overlap-(K[i]-1)) is less than or equal to 0, the calculation can be performed according to the calculation steps of the conventional convolutional layer. That is, according to the left and right frame complement parameters of the i-th convolutional layer, the L[i] frame is complemented on the left side of the W[i] frame, and the R[i] frame features are complemented on the right side, and a total of (L[i]+W[i]+R[i]) frame features are obtained as input; the (L[i]+W[i]+R[i]) frame features are input into the i-th convolutional layer, and the (L[i]+W[i]+R[i])-(K[i]-1) frame output is obtained.

[0059] It can be seen that when each convolutional layer in the convolutional neural network processes a certain window, each convolutional layer can directly copy part of the repeated results from the feature processing results of the previous window, without the need to repeatedly calculate the repeated features, thereby avoiding the waste of computing resources and improving the processing efficiency of audio data.

[0060] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a feature processing device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0061] like Figure 3 As shown, the feature processing device 300 of this embodiment includes: a windowing unit 301, configured to perform frame windowing processing on the audio data and extract the audio frame features of each window; a first processing unit 302, configured to input the audio frame features of the first window into the convolutional neural network to obtain the feature processing result of the first window; a second processing unit 303, configured to take each of the remaining windows as the target window in turn, and perform the following steps: determine the repeated frame features and non-repeated frame features of the target window relative to the previous window; determine the feature processing results of the non-repeated frame features based on the convolutional neural network, extract the feature processing results of the repeated frame features from the feature processing results of the previous window, and determine the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features.

[0062] In some optional implementations of this embodiment, the above-mentioned convolutional neural network includes at least one convolutional layer, and the above-mentioned feature processing result includes the convolution calculation result of each convolutional layer in the above-mentioned convolutional neural network.

[0063] In some optional implementations of the present embodiment, the first processing unit 302 is further configured to: input the audio frame features of the first window into the first convolutional layer of the convolutional neural network to obtain the convolution calculation result of the first convolutional layer for the first window; for each convolutional layer starting from the second convolutional layer, input the convolution calculation result of the previous convolutional layer for the first window into the convolutional layer to obtain the convolution calculation result of the convolutional layer for the first window.

[0064] In some optional implementations of the present embodiment, the second processing unit 303 is further configured to: divide the input features of each window in each convolutional layer into a first part, a second part and a third part based on a preset window length, a preset window sliding step size and a convolution kernel size in each convolutional layer in the convolutional neural network; use the second part of the input features of the target window in each convolutional layer as a repeated frame feature of the target window relative to the previous window; use the first part and the third part of the input features of the target window in each convolutional layer as non-repeated frame features of the target window relative to the previous window.

[0065] In some optional implementations of the present embodiment, the second processing unit 303 is further configured to: for each convolution layer of the convolutional neural network, input the first part and the third part of the input features of the target window in the convolution layer into the convolution layer respectively, and obtain the first convolution calculation result and the third convolution calculation result respectively; extract the convolution calculation result corresponding to the second part of the input features of the previous window in the convolution layer from the convolution calculation result of the convolution layer for the previous window as the second convolution calculation result; summarize the first convolution calculation result, the second convolution calculation result and the third convolution calculation result to obtain the convolution calculation result of the convolution layer for the target window.

[0066] In some optional implementations of the present embodiment, the second processing unit 303 is further configured to: for each convolution layer of the convolution neural network, based on the first frame complement parameter of the convolution layer, perform frame complementation on the left side of the first part of the input features of the convolution layer of the target window to obtain a first complement audio frame feature; input the first complement audio frame feature into the convolution layer to obtain a first convolution calculation result; for each convolution layer of the convolution neural network, based on the second frame complement parameter of the convolution layer, perform frame complementation on the right side of the third part of the input features of the convolution layer of the target window to obtain a second complement audio frame feature; input the second complement audio frame feature into the convolution layer to obtain a third convolution calculation result.

[0067] In some optional implementations of this embodiment, the input features of the first convolutional layer include the audio frame features of the window; the input features of the remaining convolutional layers include the convolution calculation results output by the previous convolutional layer.

[0068] The device provided by the above-mentioned embodiment of the present application performs frame and window processing on the audio data, and extracts the audio frame features of each window; then the audio frame features of the first window are input into the convolutional neural network to obtain the feature processing result of the first window; finally, each of the remaining windows is taken as the target window in turn, and the repeated frame features and non-repeated frame features of the target window relative to the previous window are determined, and the feature processing results of the non-repeated frame features are determined based on the convolutional neural network, and the feature processing results of the repeated frame features are extracted from the feature processing results of the previous window, and the feature processing results of the target window are determined based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features. Therefore, when performing feature processing on each window, the processing results of the repeated frame features of the previous window can be directly copied, and only the non-repeated frame features are processed, thereby avoiding repeated calculations of the same audio frame features, thereby avoiding waste of computing resources and improving the processing efficiency of audio data.

[0069] Figure 4 4 is a block diagram of an apparatus 400 for processing features according to an exemplary embodiment, and the apparatus 400 may be an intelligent terminal or a server. For example, the apparatus 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0070] Reference Figure 4 , the device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0071] The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-described method. In addition, the processing component 402 may include one or more modules to facilitate interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate interaction between the multimedia component 408 and the processing component 402.

[0072] The memory 404 is configured to store various types of data to support operations on the device 400. Examples of such data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0073] The power supply component 406 provides power to the various components of the device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 400.

[0074] The multimedia component 408 includes a screen that provides an output interface between the above-mentioned device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The above-mentioned touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the above-mentioned touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0075] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the device 400 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0076] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0077] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the device 400. For example, the sensor assembly 414 can detect the open / closed state of the device 400, the relative positioning of the components, such as the display and keypad of the device 400, the sensor assembly 414 can also detect the position change of the device 400 or a component of the device 400, the presence or absence of user contact with the device 400, the orientation or acceleration / deceleration of the device 400 and the temperature change of the device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0078] The communication component 416 is configured to facilitate wired or wireless communication between the device 400 and other devices. The device 400 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0079] In an exemplary embodiment, the apparatus 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0080] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of the device 400 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0081] Figure 55 is a schematic diagram of the structure of the server in some embodiments of the present application. The server 500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 522 (for example, one or more processors) and a memory 532, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 542 or data 544. Among them, the memory 532 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 522 can be configured to communicate with the storage medium 530 and execute a series of instruction operations in the storage medium 530 on the server 500.

[0082] The server 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input and output interfaces 558, one or more keyboards 556, and / or one or more operating systems 541, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0083] A non-temporary computer-readable storage medium, when the instructions in the above storage medium are executed by the processor of a device (smart terminal or server), the device is able to execute a feature processing method, the above method comprising: performing frame and window processing on audio data, and extracting audio frame features of each window; inputting the audio frame features of the first window into a convolutional neural network to obtain the feature processing results of the first window; taking each of the remaining windows as a target window in turn, and executing the following steps: determining the repeated frame features and non-repeated frame features of the target window relative to the previous window; determining the feature processing results of the non-repeated frame features based on the convolutional neural network, extracting the feature processing results of the repeated frame features from the feature processing results of the previous window, and determining the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features.

[0084] Optionally, the convolutional neural network includes at least one convolutional layer, and the feature processing result includes the convolution calculation result of each convolutional layer in the convolutional neural network.

[0085] Optionally, the step of inputting the audio frame features of the first window into a convolutional neural network to obtain a feature processing result of the first window includes: inputting the audio frame features of the first window into a first convolutional layer of the convolutional neural network to obtain a convolution calculation result of the first convolution layer for the first window; and for each convolution layer starting from the second convolutional layer, inputting the convolution calculation result of the previous convolutional layer for the first window into the convolutional layer to obtain a convolution calculation result of the convolutional layer for the first window.

[0086] Optionally, determining the repeated frame features and non-repeated frame features of the target window relative to the previous window includes: based on a preset window length, a preset window sliding step size, and a convolution kernel size in each convolution layer in the convolutional neural network, dividing the input features of each window in each convolution layer into a first part, a second part, and a third part; using the second part of the input features of the target window in each convolution layer as the repeated frame features of the target window relative to the previous window; and using the first part and the third part of the input features of the target window in each convolution layer as the non-repeated frame features of the target window relative to the previous window.

[0087] Optionally, the feature processing result of the non-repetitive frame feature is determined based on the convolutional neural network, the feature processing result of the repeated frame feature is extracted from the feature processing result of the previous window, and the feature processing result of the target window is determined based on the feature processing result of the non-repetitive frame feature and the feature processing result of the repeated frame feature, including: for each convolution layer of the convolutional neural network, the first part and the third part of the input feature of the target window in the convolution layer are respectively input to the convolution layer to obtain a first convolution calculation result and a third convolution calculation result, respectively; from the convolution calculation result of the convolution layer for the previous window, the convolution calculation result corresponding to the second part of the input feature of the previous window in the convolution layer is extracted as the second convolution calculation result; the first convolution calculation result, the second convolution calculation result and the third convolution calculation result are summarized to obtain the convolution calculation result of the convolution layer for the target window.

[0088] Optionally, for each convolution layer of the convolutional neural network, the first part and the third part of the input features of the target window in the convolutional layer are respectively input into the convolutional layer to obtain a first convolution calculation result and a third convolution calculation result, respectively, including: for each convolution layer of the convolutional neural network, based on the first frame complement parameter of the convolutional layer, the left side of the first part of the input features of the convolutional layer of the target window is complemented to obtain a first complementary audio frame feature; the first complementary audio frame feature is input into the convolutional layer to obtain a first convolution calculation result; for each convolution layer of the convolutional neural network, based on the second frame complement parameter of the convolutional layer, the right side of the third part of the input features of the convolutional layer of the target window is complemented to obtain a second complementary audio frame feature; the second complementary audio frame feature is input into the convolutional layer to obtain a third convolution calculation result.

[0089] Optionally, the input features of the first convolutional layer include the audio frame features of the window; the input features of the remaining convolutional layers include the convolution calculation results output by the previous convolutional layer.

[0090] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0091] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0092] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0093] The feature processing method, device and device for processing features provided by the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A feature processing method, characterized in that: The method comprises: Perform frame and window processing on the audio data, and extract the audio frame features of each window; Inputting the audio frame features of the first window into the convolutional neural network to obtain the feature processing result of the first window; Use each of the remaining windows as the target window in turn and perform the following steps: Determine the repeated frame features and the non-repeated frame features of the target window relative to the previous window; The feature processing result of the non-repetitive frame feature is determined based on the convolutional neural network, the feature processing result of the repeated frame feature is extracted from the feature processing result of the previous window, and the feature processing result of the target window is determined based on the feature processing result of the non-repetitive frame feature and the feature processing result of the repeated frame feature.

2. The method according to claim 1, characterized in that The convolutional neural network includes at least one convolutional layer, and the feature processing result includes the convolution calculation result of each convolutional layer in the convolutional neural network.

3. The method according to claim 2, characterized in that The step of inputting the audio frame features of the first window into the convolutional neural network to obtain the feature processing result of the first window includes: Inputting the audio frame features of the first window into the first convolution layer of the convolutional neural network to obtain the convolution calculation result of the first convolution layer for the first window; For each convolution layer starting from the second convolution layer, the convolution calculation result of the previous convolution layer for the first window is input into the convolution layer to obtain the convolution calculation result of the convolution layer for the first window.

4. The method according to claim 2, characterized in that: The determining of the repeated frame features and the non-repeated frame features of the target window relative to the previous window includes: Based on a preset window length, a preset window sliding step size, and a convolution kernel size in each convolution layer in the convolution neural network, the input features of each window in each convolution layer are divided into a first part, a second part, and a third part; Using the second part of the target window in the input features of each convolutional layer as the repeated frame feature of the target window relative to the previous window; The first part and the third part of the target window in the input features of each convolutional layer are used as the non-repeated frame features of the target window relative to the previous window.

5. The method according to claim 4, characterized in that The step of determining the feature processing result of the non-repeated frame feature based on the convolutional neural network, extracting the feature processing result of the repeated frame feature from the feature processing result of the previous window, and determining the feature processing result of the target window based on the feature processing result of the non-repeated frame feature and the feature processing result of the repeated frame feature, comprises: For each convolution layer of the convolutional neural network, the first part and the third part of the input features of the target window in the convolution layer are respectively input into the convolution layer to obtain a first convolution calculation result and a third convolution calculation result, respectively; from the convolution calculation result of the convolution layer for the previous window, the convolution calculation result corresponding to the second part of the input features of the previous window in the convolution layer is extracted as the second convolution calculation result; the first convolution calculation result, the second convolution calculation result and the third convolution calculation result are summarized to obtain the convolution calculation result of the convolution layer for the target window.

6. The method according to claim 5, characterized in that For each convolution layer of the convolutional neural network, the first part and the third part of the input features of the target window in the convolution layer are respectively input to the convolution layer to obtain a first convolution calculation result and a third convolution calculation result respectively, including: For each convolution layer of the convolution neural network, based on a first frame-padding parameter of the convolution layer, padding the left side of the first part of the input feature of the convolution layer to obtain a first compensated audio frame feature; inputting the first compensated audio frame feature to the convolution layer to obtain a first convolution calculation result; For each convolution layer of the convolutional neural network, based on the second frame complement parameter of the convolution layer, the target window is frame-padded on the right side of the third part of the input feature of the convolution layer to obtain a second complementary audio frame feature; the second complementary audio frame feature is input into the convolution layer to obtain a third convolution calculation result.

7. The method according to claim 4, characterized in that The input features of the first convolutional layer include the audio frame features of the window; the input features of the remaining convolutional layers include the convolution calculation results output by the previous convolutional layer.

8. A feature processing device, characterized in that: The device comprises: A windowing unit is configured to perform frame-by-frame windowing processing on the audio data and extract audio frame features of each window; A first processing unit is configured to input the audio frame features of the first window into a convolutional neural network to obtain a feature processing result of the first window; The second processing unit is configured to sequentially use each of the remaining windows as a target window and perform the following steps: Determine the repeated frame features and non-repeated frame features of the target window relative to the previous window; determine the feature processing results of the non-repeated frame features based on the convolutional neural network, extract the feature processing results of the repeated frame features from the feature processing results of the previous window, and determine the feature processing results of the target window based on the feature processing results of the non-repeated frame features and the feature processing results of the repeated frame features.

9. The device according to claim 8, characterized in that The convolutional neural network includes at least one convolutional layer, and the feature processing result includes the convolution calculation result of each convolutional layer in the convolutional neural network.

10. The device according to claim 9, characterized in that The first processing unit is further configured to: Inputting the audio frame features of the first window into the first convolution layer of the convolutional neural network to obtain the convolution calculation result of the first convolution layer for the first window; For each convolution layer starting from the second convolution layer, the convolution calculation result of the previous convolution layer for the first window is input into the convolution layer to obtain the convolution calculation result of the convolution layer for the first window.

11. The device according to claim 9, characterized in that The second processing unit is further configured to: Based on a preset window length, a preset window sliding step size, and a convolution kernel size in each convolution layer in the convolution neural network, the input features of each window in each convolution layer are divided into a first part, a second part, and a third part; Using the second part of the target window in the input features of each convolutional layer as the repeated frame feature of the target window relative to the previous window; The first part and the third part of the target window in the input features of each convolutional layer are used as the non-repeated frame features of the target window relative to the previous window.

12. The device according to claim 11, characterized in that The second processing unit is further configured to: For each convolution layer of the convolutional neural network, the first part and the third part of the input features of the target window in the convolution layer are respectively input into the convolution layer to obtain a first convolution calculation result and a third convolution calculation result, respectively; from the convolution calculation result of the convolution layer for the previous window, the convolution calculation result corresponding to the second part of the input features of the previous window in the convolution layer is extracted as the second convolution calculation result; the first convolution calculation result, the second convolution calculation result and the third convolution calculation result are summarized to obtain the convolution calculation result of the convolution layer for the target window.

13. The device according to claim 12, characterized in that The second processing unit is further configured to: For each convolution layer of the convolutional neural network, based on a first frame-padding parameter of the convolution layer, padding the left side of a first part of the input feature of the convolution layer of the target window to obtain a first padded audio frame feature; Inputting the first complementary audio frame feature into the convolution layer to obtain a first convolution calculation result; For each convolution layer of the convolution neural network, based on the second frame-padding parameter of the convolution layer, padding the target window on the right side of the third part of the input feature of the convolution layer to obtain a second padding audio frame feature; The second complementary audio frame feature is input into the convolution layer to obtain a third convolution calculation result.

14. A device for processing features, characterized in that The invention comprises a memory and one or more programs, wherein the one or more programs are stored in the memory, and when the programs are executed by one or more processors, the steps of any method described in claims 1-7 are implemented.

15. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images