Sound data classification method, terminal and medium based on multi-dimensional feature weighted fusion

Abstract

The present invention discloses a sound data classification method, terminal and medium based on multi-dimensional feature weighted fusion, the method comprising the following steps: S1, pre-processing the input sound data to obtain sound segment data; S2, extracting static and dynamic combined features J from the sound segment data; S3, extracting initial input features K from the sound segment data; S4, processing the static and dynamic combined features J and the initial input features K through a fully connected neural network to obtain an attention weight vector H; S5, calculating the final input feature G, G=J*H+K*(1-H); S6, inputting the final input feature G into an LSTM network to obtain a category m of sound data classification. The present invention links the front and back frames together by adding dynamic differential features, and at the same time, redistributes the resources evenly distributed on different dimensions of the fused features according to the attention weight, so as to achieve the purpose of "learning from each other's strengths and making up for each other's weaknesses", making the fused features more effective and more distinguishable, thereby improving the accuracy of classification.

Description

Technical Field

[0001] The present invention relates to the field of sound data processing, and in particular to a sound data classification method based on multi-dimensional feature weighted fusion, a terminal and a computer-readable storage medium. Background Art

[0002] Based on the vehicle sound data collected by the acoustic array, the field vehicle target is identified. Classification and identification is generally divided into two steps: first, extract the characteristics of the target sound signal; second, design a classifier to obtain the discrimination result. Commonly used sound signal features include: Linear Prediction Coefficient (LPC), Mel-Frequency Cepstral Coefficient (MFCC) and zero-crossing rate (ZCR).

[0003] On the other hand, deep learning is currently in its infancy, and it is also common to extract features through neural networks. Based on the above-mentioned primary form features, the classification model generally performs multi-feature fusion to obtain fusion features with stronger expression capabilities. There are generally two ways to fuse features in the existing technology: (1) feature concatenation as fusion features, the feature dimension becomes twice the original; (2) corresponding dimension addition as fusion features, the feature dimension remains unchanged. Multi-feature fusion is widely used in language recognition, speaker recognition, voiceprint recognition and other fields, and has been successfully applied to the classification and recognition of vehicle targets in recent years. However, these two feature fusion methods default to equal contribution values ​​for their respective features, do not highlight more effective feature components, and do not suppress other useless feature components, which may cause dimensionality disaster and fail to achieve the expected goal of feature fusion (i.e. "taking advantage of others' strengths and making up for their weaknesses"), affecting the feature fusion results. Summary of the invention

[0004] The purpose of the present invention is to provide a sound data classification method, terminal and storage medium based on multi-dimensional feature weighted fusion to solve the above problems. To this end, the technical solution adopted by the present invention is as follows:

[0005] According to one aspect of the present invention, a method for classifying sound data based on weighted fusion of multi-dimensional features is provided, comprising the following steps:

[0006] S1, preprocessing the input sound data to obtain sound segment data;

[0007] S2, extracting static and dynamic combined features J from the sound segment data;

[0008] S3, extracting initial input features K from the sound segment data;

[0009] S4, the static and dynamic combined features J and the initial input features K are processed by a fully connected neural network to obtain the attention weight vector H;

[0010] S5. Calculate the final input feature G, G = J*H + K*(1-H);

[0011] S6. Input the final input feature G into the LSTM network to obtain the category m of the sound data classification.

[0012] Furthermore, the specific process of S1 is:

[0013] S11, downsampling the sound data of channel p by a times, wherein the length of each frame of data is (1 / a)*fs, wherein fs is the sampling frequency of the sound data;

[0014] S12, randomly select a channel data as input;

[0015] S13, cutting the channel data into b-frame segments to obtain the sound segment data, wherein the sound segment data is segmented into 1 frame.

[0016] Further, a is 8 or 16, b=5-50, and fs is 22kHz, 44kHz or 88kHz.

[0017] Furthermore, the specific process of S2 is:

[0018] S21, removing the first and last two frames of each cut sound segment data, and extracting a total of (b-2)*N dimensional static features therefrom;

[0019] S22, extracting n-order dynamic differential features from the above static features, and also generating (b-2)*N-dimensional dynamic differential features;

[0020] S23, combining the static and dynamic features together, that is, generating the static and dynamic combined feature J of (b-2)*2*N dimensions.

[0021] Further, N=32, specifically, the 32-dimensional static features include 8-dimensional linear prediction coefficients, 23-dimensional Mel-frequency cepstral coefficients and 1-dimensional zero-crossing rate.

[0022] Furthermore, the specific process of S3 is: sending the b-2 frame of the sound segment data in the b frame into the deep learning network to generate the initial input feature K of (b-2)*2*N dimensions.

[0023] Furthermore, the network structure of the deep learning network is an 8-layer convolutional neural network, with 3 convolution kernels and a step size of 2 or 1. Batch normalization BatchNorm1d, activation function relu and maximum pooling MaxPool1d are added after each layer of convolution.

[0024] Furthermore, the specific process of S4 is:

[0025] S41, by feature concatenation, the (b-2)*2*N dimensional static and dynamic combined feature J and the (b-2)*2*N dimensional initial input feature K are concatenated into a (b-2)*4*N dimensional combined feature;

[0026] S42, through a 3-layer fully connected neural network, the (b-2)*4*N-dimensional combined features are reduced to (b-2)*2*N dimensions;

[0027] S43, through the softmax layer, normalized to between 0 and 1, forming a (b-2)*2*N-dimensional attention weight vector H;

[0028] Among them, the numbers of hidden layer neurons in the first layer fully connected network, the second layer fully connected network and the third layer fully connected network F3 are 4*N, 3*N and 2*N respectively.

[0029] According to another aspect of the present invention, a terminal is provided, comprising a processor and a memory, wherein the memory stores program instructions, wherein the processor executes the program instructions to implement the steps of the method as claimed above.

[0030] According to yet another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores program instructions executable by a processor to implement the steps of the method described above.

[0031] The present invention connects the previous and next frames by adding dynamic differential features, so that the dynamic features can complement the static features, making the combination of static and dynamic features have better feature distinguishability. At the same time, the resources evenly distributed on different dimensions of the fused features are redistributed according to the attention weights. The features with large weights are important features, and the important features are given more attention, while the others are given less attention. This achieves the purpose of "taking advantage of each other's strengths and overcoming each other's weaknesses", making the fused features more effective and more distinguishable, thereby improving the accuracy of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of a sound data classification method based on multi-dimensional feature weighted fusion according to an embodiment of the present invention;

[0033] Figure 2 It is a schematic diagram of attention weight vector generation in a sound data classification method based on multi-dimensional feature weighted fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings so that the purpose, features and advantages of the present invention can be more clearly understood. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0035] In the following description, certain specific details are set forth for the purpose of illustrating the various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0036] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0037] In the following description, in order to clearly show the structure and working mode of the present invention, many directional words will be used for description, but the words "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", "down", etc. should be understood as convenient terms and should not be understood as restrictive terms.

[0038] like Figure 1 As shown, the first embodiment of the present invention provides a sound data classification method based on multi-dimensional feature weighted fusion, comprising the following steps:

[0039] S1. Preprocess the input sound data to obtain sound segment data. The specific process is as follows:

[0040] S11, the sound data of the p channel is downsampled a times, wherein the length of each frame of data is (1 / a)*fs, wherein p≥1, and fs is the sampling frequency of the sound data. Here, the sound data is specifically the vehicle sound data collected by the uniform circular array. In order to make the vehicle sound data usable for different purposes, the sampling frequency is usually relatively high, for example, the sampling frequency can be 22kHz, 44kHz or 88kHz. However, for the classification purposes of the present application, the sampling frequency is too high, the amount of data is too large, and it is not conducive to data processing. Therefore, it is necessary to perform a-fold downsampling in advance to reduce the amount of data. The value of a can be selected according to the actual situation, such as 8 or 16.

[0041] S12. Randomly select a channel data as input.

[0042] S13, the channel data is cut into segments of b frames to obtain the sound segment data, wherein the segment of the sound segment data is 1 frame and the segment length is b frames. The value of b is selected according to the implementation situation, and can usually be selected between 5 and 50.

[0043] S2, extracting static and dynamic combined features J from the sound segment data. The specific process is as follows:

[0044] S21, remove the first and last two frames of each cut sound segment data, and extract a total of (b-2)*N dimensional static features therefrom. In a specific embodiment, 8-dimensional linear prediction coefficients LPC, 23-dimensional Mel-frequency cepstral coefficients MFCC and 1-dimensional zero crossing rate ZCR are extracted from each cut sound segment data, totaling (b-2)*32 dimensional static features. The following is an example of N=32.

[0045] S22. Extract n-order dynamic differential features from the above static features, and also generate (b-2)*32-dimensional dynamic differential features.

[0046] S23. Combine the static and dynamic features together to generate a (b-2)*64-dimensional static and dynamic combined feature J.

[0047] Since the first and last frames of the dynamic difference are zero values ​​and meaningless, the data of the first and last frames in the sound segment data are discarded. At the same time, since the segment length is b times the segment shift difference, there is a large amount of repeated data, so there will be no data loss. The static feature only operates on the data of a certain frame without considering the timing of the vehicle signal. Therefore, this method adds dynamic differential features to connect the previous and next frames. Dynamic features can complement static features, so that the combination of static and dynamic features can have better feature differentiation.

[0048] S3. Extract the initial input feature K from the sound segment data. Specifically, the input is data of (1 / a)*fs sampling points, and the b-2 frames in the b-frame sound segment data are input into the deep learning network, and the (b-2)*64-dimensional initial input feature K is generated through an 8-layer convolutional neural network. Among them, the network structure of the deep learning network is an 8-layer convolutional neural network, with a convolution kernel of 3 and a step size of 2 or 1. Batch normalization BatchNorm1d, activation function relu and maximum pooling MaxPool1d are added after each convolution layer.

[0049] S4. Process the static and dynamic combined features J and the initial input features K through a fully connected neural network to obtain the attention weight vector H. The specific process is as follows:

[0050] S41, through feature concatenation, the (b-2)*64-dimensional static and dynamic combined feature J and the (b-2)*64-dimensional initial input feature K are concatenated into a (b-2)*128-dimensional combined feature;

[0051] S42, through a 3-layer fully connected neural network, the (b-2)*128-dimensional combined features are reduced to (b-2)*64 dimensions;

[0052] S43, through the softmax layer, normalized to between 0 and 1, forming a (b-2)*64-dimensional attention weight vector H.

[0053] Figure 2 The process of generating the attention weight vector H is shown. The intermediate process of generating the weight forms the following form:

[0054] Y1=F1(J,K)

[0055] Y2=F2(Y1)

[0056] Y3=F3(Y2)

[0057] The input of the neural network is the static and dynamic combined feature J and the initial input feature K, and then passes through the first layer of fully connected network F1, the number of hidden layer neurons is 128, and the output is Y1; the second layer of fully connected network F2, the input is Y1, the number of hidden layer neurons is 96, the output is Y2, the third layer of fully connected network F3, the input is Y2, the number of hidden layer neurons is 64, and the output is Y3. Y3 is passed to the output layer with softmax and normalized to between 0 and 1 to get the current H. The calculation formula is as follows:

[0058] H = softmax(Y3)

[0059] S5. Calculate the final input feature G, G = J*H + K*(1-H). The key to learning the weights of the feature components through the attention mechanism is to find the correlation between them based on the original data and highlight some important feature dimensions in the fusion feature dimension. The purpose is to obtain as much detailed effective information as possible and reduce the influence of irrelevant and useless information; enhance the distinguishability of the extracted fusion features, so that the fused features can focus more on certain feature dimension components.

[0060] S6. Input the final input feature G into the LSTM network to obtain the category m of the sound data classification. Since the dynamic differential feature connects the previous and next frames, and the vehicle sound signal is related in time sequence, a deep learning network with time sequence should be used. Therefore, this method uses a long short-term memory network (LSTM) as the classification network, specifically a 2-layer LSTM network, with 32 hidden neurons in the first layer and 16 hidden neurons in the second layer, and the output is the classification category m.

[0061] The present invention connects the previous and next frames by adding dynamic differential features, so that the dynamic features can complement the static features, making the combination of static and dynamic features have better feature distinguishability. At the same time, the resources evenly distributed on different dimensions of the fused features are redistributed according to the attention weights. The features with large weights are important features, and the important features are given more attention, while the others are given less attention. This achieves the purpose of "taking advantage of each other's strengths and overcoming each other's weaknesses", making the fused features more effective and more distinguishable, thereby improving the accuracy of classification.

[0062] A second embodiment of the present invention provides a terminal, comprising a processor and a memory, wherein the memory stores program instructions, wherein the processor executes the program instructions to implement steps S1-S6 of the method as claimed above.

[0063] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program in the terminal.

[0064] The terminal may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The terminal may include, but is not limited to, a processor and a memory. For example, it may also include input and output devices, network access devices, buses, etc.

[0065] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0066] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the terminal device by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0067] A third embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions executable by a processor to implement steps S1-S6 of the above method.

[0068] If each module / unit of the terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0069] Preferred embodiments of the present invention have been described above in detail, but it should be understood that aspects of the embodiments can be modified, if necessary, to employ aspects, features and concepts of the various patents, applications and publications to provide further embodiments.

[0070] These and other changes can be made to the embodiments in light of the above detailed description.In general, in the claims, the terms used should not be considered limited to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which these claims are entitled.

Claims

1. A sound data classification method based on multi-dimensional feature weighted fusion, characterized in that: The following steps are involved: S1. Preprocess the input sound data to obtain sound segment data. The specific process is as follows: S11, downsampling the sound data of channel p by a times, wherein the length of each frame of data is (1 / a)*fs, wherein fs is the sampling frequency of the sound data; S12, randomly select a channel data as input; S13, cutting the channel data into b frames to obtain the sound segment data; S2. Extract static and dynamic combined features J from the sound segment data. The specific process is as follows: S21, removing the first and last two frames of each cut sound segment data, and extracting a total of (b-2)*N dimensional static features therefrom; S22, extracting n-order dynamic differential features from the above static features, and also generating (b-2)*N-dimensional dynamic differential features; S23, combining the static and dynamic features together, that is, generating the static and dynamic combined feature J of (b-2)*2*N dimensions; S3, extracting the initial input feature K from the sound segment data, the specific process is: sending b-2 frames of the sound segment data in b frames into the deep learning network to generate the initial input feature K of (b-2)*2*N dimensions; S4. The static and dynamic combined features J and the initial input features K are processed by a fully connected neural network to obtain the attention weight vector H. The specific process is as follows: S41, by feature concatenation, the (b-2)*2*N dimensional static and dynamic combined feature J and the (b-2)*2*N dimensional initial input feature K are concatenated into a (b-2)*4*N dimensional combined feature; S42, through a 3-layer fully connected neural network, the (b-2)*4*N-dimensional combined features are reduced to (b-2)*2*N dimensions; S43, through the softmax layer, normalized to between 0 and 1, forming a (b-2)*2*N-dimensional attention weight vector H; S5. Calculate the final input feature G, G = J*H + K*(1-H); S6. Input the final input feature G into the LSTM network to obtain the category m of the sound data classification.

2. The method according to claim 1, characterized in that The segment shift of the sound segment data is 1 frame.

3. The method according to claim 2, characterized in that a is 8 or 16, b=5-50, and fs is 22k, 44k or 88k.

4. The method according to claim 1, characterized in that N=32. Specifically, the 32-dimensional static features include 8-dimensional linear prediction coefficients, 23-dimensional Mel-frequency cepstral coefficients, and 1-dimensional zero-crossing rate.

5. The method according to claim 1, characterized in that The network structure of the deep learning network is an 8-layer convolutional neural network with 3 convolution kernels and a step size of 2 or 1. Batch normalization BatchNorm1d, activation function relu and maximum pooling MaxPool1d are added after each convolution layer.

6. The method according to claim 1, characterized in that The number of hidden layer neurons in the first layer fully connected network, the second layer fully connected network and the third layer fully connected network F3 are 4*N, 3*N and 2*N respectively.

7. A terminal comprising a processor and a memory, wherein the memory stores program instructions, characterized in that: The processor executes the program instructions to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions executable by a processor to implement the steps of the method according to any one of claims 1 to 6.

Images