Complete time sequence feature extraction method for environmental sound recognition
By performing multi-step processing of ambient sound signals and semantic representation subspace learning, high-dimensional, redundant-free semantic representation is generated, which solves the problem of insufficient timing information in the prior art, and significantly improves the performance and efficiency of ambient sound recognition.
Patent Information
- Application Number
- CN202411270581.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The timing information obtained by the existing methods contains limited timing information, resulting in poor environmental sound recognition performance, and a new method for extracting timing characteristics of ambient sound signal is needed to improve recognition performance.
Using a complete timing feature extraction method for environmental sound recognition, the ambient sound signal is preprocessed, framed, segmented and multiple post-processed, and the complete semantic representation subspace under timing constraints is learned to generate high-dimensional and non-existent Redundant complete semantics represent subspaces and input them into the support vector machine for identification.
The obtained complete semantic representation subspace contains sufficient timing information, which significantly improves the performance and recognition efficiency of environmental sound recognition. At the same time, the model structure is simple and the demand for software and hardware resources is low.
Smart Images

Figure CN119049456A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of time series feature extraction for environmental sound signal recognition, and in particular relates to a complete time series feature extraction method for environmental sound recognition. Background Art
[0002] With the rapid development of deep learning technology, the recognition of environmental sounds in combination with artificial intelligence technology has become a hot topic among researchers in recent years. At present, most research works use traditional machine learning models to characterize the content information of environmental sounds. A small number of research works characterize the corresponding time series information and verify that time series information is very important for improving the performance of environmental sound recognition. However, existing works can often only obtain time series representations with low feature dimensions, and the time series representations of low-dimensional features contain limited time series information, which makes it difficult to fully characterize complex environmental sound signals.
[0003] In summary, since the time series representation obtained by the existing method contains limited time series information, the performance of environmental sound recognition based on the time series representation obtained by the existing method is still poor, and the performance of environmental sound recognition needs to be further improved. It is very necessary to propose a new method for extracting time series features of environmental sound signals. Summary of the invention
[0004] The purpose of the present invention is to solve the problem of limited timing information contained in the timing representation obtained by the existing method, and to propose a complete timing feature extraction method for environmental sound recognition, and then perform environmental sound recognition based on the extracted timing features, which can improve the performance of environmental sound recognition.
[0005] The technical solution adopted by the present invention to solve the above technical problems is: a complete time series feature extraction method for environmental sound recognition, the method specifically comprising the following steps:
[0006] Step 1: preprocessing the acquired ambient sound signal to obtain a preprocessed ambient sound signal;
[0007] Step 2: The pre-processed ambient sound signal is subjected to frame processing, and then features are extracted for each frame obtained by the frame processing, and the frame-level feature representation sequence of the pre-processed ambient sound signal is formed by using the extracted features of each frame;
[0008] Step 3: segment the frame processing result of the preprocessed ambient sound signal, extract the features of each audio segment respectively, and use the features of each audio segment to form a segment-level feature representation sequence of the preprocessed ambient sound signal;
[0009] Step 4: post-processing the segment-level feature representation sequence of the pre-processed ambient sound signal to obtain a final segment-level feature representation sequence of the pre-processed ambient sound signal;
[0010] The post-processing includes a first post-processing, a second post-processing and a third post-processing;
[0011] Step 5: learning a complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the preprocessed ambient sound signal, and the obtained complete semantic representation subspace is the complete temporal feature;
[0012] The specific process of step five is as follows:
[0013] The objective function of learning a complete semantic representation subspace under temporal constraints is expressed as:
[0014]
[0015] st,(D C ) T a n <(D k ) T a m
[0016] Where A is the final segment-level feature representation sequence of the ambient sound signal after preprocessing, D∈R d×p is the basis matrix of a subspace of dimension p, where p is the pre-specified subspace dimension, I p×p is the identity matrix of dimension p×p, D T is the transposed matrix of D, D k is the kth column vector in the basis matrix D, a n with a m are the features of the nth and mth audio segments in A respectively, and n <m,‖·‖ F represents the F-norm;
[0017] The objective function of formula (3) is equivalently converted to:
[0018]
[0019] Among them, S(d,p) is a Stiefel manifold composed of all matrices of dimension d×p that satisfy the column orthogonality relation;
[0020] Then calculate the manifold S(d,p) so that the objective function The matrix D with the smallest value is taken as the complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the preprocessed ambient sound signal.
[0021] Furthermore, the segment-level feature representation sequence of the pre-processed ambient sound signal is post-processed, and the specific process is as follows:
[0022] The segment-level feature representation sequence of the preprocessed ambient sound signal is recorded as yes The feature representation of the nth audio segment in , n = 1, 2, ..., N, N is the total number of audio segments corresponding to the ambient sound signal after preprocessing, d is The dimension of , R is a real number, Indicated by The sequence of components;
[0023] right Perform a first post-processing to obtain a first post-processing result; perform a second post-processing on the first post-processing result to obtain a second post-processing result; perform a third post-processing on the second post-processing result, and use the third post-processing result as the final segment-level feature representation sequence of the pre-processed ambient sound signal.
[0024] Furthermore, the specific process of the first post-processing is:
[0025]
[0026] in, It is the feature representation of the nth audio segment in the first post-processing result.
[0027] Furthermore, the specific process of the second post-processing is:
[0028]
[0029] in, is the feature representation of the nth audio segment in the second post-processing result, and ‖·‖ represents the 2-norm.
[0030] Furthermore, the specific process of the third post-processing is:
[0031]
[0032] Among them, |·| represents taking the absolute value, sign(·) represents the sign function, and a n is the feature representation of the nth audio segment in the final segment-level feature representation sequence;
[0033] The final segment-level feature representation sequence of the preprocessed ambient sound signal is recorded as A = [a n ],[a n ] indicates that a 1 ,
[0034] a 2, …, a N The sequence composed of.
[0035] Furthermore, the complete semantic representation subspace is used to identify the acquired environmental sound signal, and the specific method of identification is:
[0036] The complete semantic representation subspace is used as the input of a support vector machine, and the support vector machine outputs the recognition result of the acquired environmental sound signal.
[0037] Furthermore, the training method of the support vector machine is:
[0038] Step 1: Obtain an environmental sound signal training data set, and preprocess each environmental sound signal in the training data set to obtain each preprocessed environmental sound signal;
[0039] Step 2: for the preprocessed i-th ambient sound signal in the training data set, perform frame processing on the preprocessed i-th ambient sound signal, and then perform feature extraction on each frame obtained by the frame processing, and use the extracted features of each frame to form a frame-level feature representation sequence of the preprocessed i-th ambient sound signal;
[0040] Where i = 1, 2, ..., I, I is the total number of environmental sound signals in the training data set;
[0041] Step 3: Segment the frame processing result of the i-th ambient sound signal after preprocessing in the training data set, extract the features of each audio segment respectively, and use the features of each audio segment to form a segment-level feature representation sequence of the i-th ambient sound signal after preprocessing;
[0042] Step 4: post-processing the segment-level feature representation sequence of the i-th ambient sound signal to obtain a final segment-level feature representation sequence of the i-th ambient sound signal;
[0043] Step 5: learning a complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the i-th ambient sound signal;
[0044] Step 6: Use the category labels of each signal in the environmental sound signal training data set and the learned complete semantic representation subspace to train the support vector machine.
[0045] The beneficial effects of the present invention are:
[0046] The method of the present invention can obtain a complete semantic representation subspace with higher feature dimensions and no redundant information. The information contained in the obtained complete semantic representation subspace is effective for completely describing the event timing information. Since the timing information between audio segments is introduced during the learning process, the complete semantic representation subspace contains sufficient timing information, thereby improving the performance of environmental sound recognition and greatly improving the recognition efficiency. Moreover, the model structure of the method of the present invention is simple and has low requirements for software and hardware resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of a complete time series feature extraction method for environmental sound recognition of the present invention;
[0048] Figure 2 It is a performance comparison chart of the method of the present invention and other methods on the ESC-50 data set. DETAILED DESCRIPTION
[0049] Specific implementation method 1: Combination Figure 1 The present embodiment describes a complete temporal feature extraction method for environmental sound recognition, and the method specifically comprises the following steps:
[0050] Step 1: preprocessing the acquired ambient sound signal to obtain a preprocessed ambient sound signal;
[0051] Among them, the preprocessing uses a sampling rate of 44100 Hz and 16-bit quantization;
[0052] Step 2: The pre-processed ambient sound signal is divided into frames, and then Mel spectrum coefficient features are extracted for each frame obtained by the frame division process, and the frame-level feature representation sequence of the pre-processed ambient sound signal is formed by using the extracted features of each frame;
[0053] The frame processing meets the following requirements: the length of each audio frame is 23 milliseconds, and the overlap length between adjacent audio frames is 11.5 milliseconds
[0054] Step 3: segment the frame processing result of the preprocessed ambient sound signal, extract the features of each audio segment, and use the features of each audio segment to form a segment-level feature representation sequence of the preprocessed ambient sound signal;
[0055] The following requirements are met during segment processing: each segment includes 84 audio frames (about 2 seconds), and the overlap length between adjacent segments is 20 frames (about 0.5 seconds). The feature extraction of audio segments is done using a pre-trained deep learning model. The parameters of the deep learning model are obtained through pre-training on a large-scale dataset and remain fixed when extracting segment-level features. No additional learning is required, and there is no corresponding learning cost.
[0056] Step 4: Post-process the segment-level feature representation sequence of the preprocessed environmental sound signal to obtain the final segment-level feature representation sequence of the preprocessed environmental sound signal;
[0057] The post-processing includes first post-processing, second post-processing, and third post-processing;
[0058] Step 5: Learn the complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the preprocessed environmental sound signal, and the obtained complete semantic representation subspace is the complete temporal feature;
[0059] The specific process of Step 5 is as follows:
[0060] In the way of minimizing the reconstruction error, the process of learning the complete semantic representation subspace is expressed as:
[0061]
[0062] where D ∈ R d×p is the basis matrix of a subspace with dimension p. p is a pre-specified subspace dimension. Generally, p can be set as a constant that is much smaller than d and greater than 1 to ensure learning a multi-dimensional semantic subspace and satisfy the constraint D T D = I p×p , and the constraint is used to ensure the orthogonality between different columns of the basis matrix, ensure that the matrix is an orthogonal matrix, and further ensure the completeness of the corresponding subspace and no redundant information;
[0063] In order to introduce temporal information when learning the complete semantic representation subspace, the present invention characterizes the overall temporal information of the segment sequence by maintaining the appearance order between every two segments. Taking any two segment-level feature representations a n and a m in A as an example, a n and a m are the nth and mth segment-level feature representations in A respectively, and n, m = 1,..., N; if n < m, it can be known that a n occurs before a m , that is, a n and a m are a pair of segment-level representations with a sequential relationship; to maintain the natural sequential relationship between a n and a m , first obtain their representations on each column vector of the semantic space basis matrix respectively, and then realize it by maintaining the magnitude relationship of these representations numerically. This temporal constraint can be specifically expressed as:
[0064] (D k ) T a n < (D k )T a m (2)
[0065] Among them, D k is the kth column vector in the basis matrix D, and k=1,2,...,p; (D k ) T a n and (D k ) T a m Respectively, segment level represents a n with a m In the basis vector D k The above representation.
[0066] Finally, semantic space learning based on minimizing reconstruction error is performed under the above-mentioned temporal constraints, and then the corresponding temporal feature representation is obtained.
[0067] The objective function of learning a complete semantic representation subspace under temporal constraints is expressed as:
[0068]
[0069] st,(D k ) T a n <(D k ) T a m
[0070] Where A is the final segment-level feature representation sequence of the ambient sound signal after preprocessing, D∈R d×p is the basis matrix of a subspace of dimension p, where p is the pre-specified subspace dimension, I p×p is the identity matrix of dimension p×p, D T is the transposed matrix of D, D k is the kth column vector in the basis matrix D, a n with a m are the features of the nth and mth audio segments in A respectively, and n <m,‖·‖ F represents the F-norm;
[0071] The objective function of formula (3) is equivalently converted to:
[0072]
[0073] Among them, S(d,p) is a Stiefel manifold composed of all matrices of dimension d×p that satisfy the column orthogonality relation;
[0074] Then calculate the manifold S(d,p) so that the objective function The matrix D with the smallest value is taken as the complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the preprocessed ambient sound signal.
[0075] Since D is a multi-dimensional orthogonal matrix, the temporal feature representation of the event does not contain redundant information, and the information contained is valid for completely characterizing the temporal information of the event; since the temporal information between audio segments is introduced in the learning process, the obtained complete semantic representation subspace contains the temporal feature representation of the signal.
[0076] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the segment-level feature representation sequence of the pre-processed ambient sound signal is post-processed, and the specific process is as follows:
[0077] The segment-level feature representation sequence of the preprocessed ambient sound signal is recorded as yes The feature representation of the nth audio segment in , n = 1, 2, ..., N, N is the total number of audio segments corresponding to the ambient sound signal after preprocessing, d is The dimension of , R is a real number, Indicated by The sequence of components;
[0078] right Perform a first post-processing to obtain a first post-processing result; perform a second post-processing on the first post-processing result to obtain a second post-processing result; perform a third post-processing on the second post-processing result, and use the third post-processing result as the final segment-level feature representation sequence of the pre-processed ambient sound signal.
[0079] The other steps and parameters are the same as those in the first embodiment.
[0080] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the specific process of the first post-processing is as follows:
[0081]
[0082] in, It is the feature representation of the nth audio segment in the first post-processing result.
[0083] The other steps and parameters are the same as those in the first or second embodiment.
[0084] Specific implementation method 4: This implementation method is different from the specific implementation methods 1 to 3 in that the specific process of the second post-processing is as follows:
[0085]
[0086] in, is the feature representation of the nth audio segment in the second post-processing result, and ‖·‖ represents the 2-norm.
[0087] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.
[0088] Specific implementation method 5: This implementation method is different from the specific implementation methods 1 to 4 in that the specific process of the third post-processing is as follows:
[0089]
[0090] Among them, |·| represents taking the absolute value, sign(·) represents the sign function, and a n is the feature representation of the nth audio segment in the final segment-level feature representation sequence;
[0091] The final segment-level feature representation sequence of the preprocessed ambient sound signal is recorded as A = [a n ],[a n ] indicates that a 1 、a 2 , …, a N The sequence composed of.
[0092] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.
[0093] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the complete semantic representation subspace is used to identify the acquired ambient sound signal, and the specific method of identification is:
[0094] The complete semantic representation subspace is used as the input of a support vector machine, and the support vector machine outputs the recognition result of the acquired environmental sound signal.
[0095] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.
[0096] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the training method of the support vector machine is:
[0097] Step 1: Obtain an environmental sound signal training data set, and preprocess each environmental sound signal in the training data set to obtain each preprocessed environmental sound signal;
[0098] The acquired training data set includes various categories of environmental sound signals to improve the generalization performance of the model;
[0099] Step 2: for the preprocessed i-th ambient sound signal in the training data set, perform frame processing on the preprocessed i-th ambient sound signal, and then perform Mel spectrum coefficient feature extraction on each frame obtained by the frame processing, and use the extracted frame features to form a frame-level feature representation sequence of the preprocessed i-th ambient sound signal;
[0100] Where i = 1, 2, ..., I, I is the total number of environmental sound signals in the training data set;
[0101] The number of Mel-spectrogram bands is set to 128, and other parameters use the default values to describe the acoustic characteristics within the image frame;
[0102] Step 3: Segment the frame processing result of the i-th ambient sound signal after preprocessing in the training data set, extract the features of each audio segment respectively, and use the features of each audio segment to form a segment-level feature representation sequence of the i-th ambient sound signal after preprocessing;
[0103] Step 4: post-processing the segment-level feature representation sequence of the i-th ambient sound signal to obtain a final segment-level feature representation sequence of the i-th ambient sound signal;
[0104] Step 5: learning a complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the i-th ambient sound signal;
[0105] Step 6: Use the category labels of each signal in the environmental sound signal training data set and the learned complete semantic representation subspace to train the support vector machine.
[0106] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.
[0107] The parameters required for support vector machine training all use the default values in the Libsvm toolkit to obtain the weight matrix of the support vector machine and the corresponding classification model. The complete semantic representation subspace corresponding to the test sample is used as the input of the trained support vector machine, and the label corresponding to the maximum matching score is selected as the prediction result. The recognition accuracy is calculated based on the similarities and differences between the predicted type and the actual type.
[0108] The present invention selects the mainstream deep learning model in the current field of environmental sound recognition, and compares its recognition performance with the recognition performance achieved by the method of the present invention. Figure 2 The figure shows the performance comparison between the method of the present invention and other related mainstream methods on the ESC-50 dataset. Figure 2 It can be seen that compared with the four comparison methods, the method of the present invention can achieve higher recognition performance.
[0109] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.
Claims
1. A complete temporal feature extraction method for environmental sound recognition, characterized in that: The method specifically comprises the following steps: Step 1: preprocessing the acquired ambient sound signal to obtain a preprocessed ambient sound signal; Step 2: The pre-processed ambient sound signal is subjected to frame processing, and then features are extracted for each frame obtained by the frame processing, and the frame-level feature representation sequence of the pre-processed ambient sound signal is formed by using the extracted features of each frame; Step 3: segment the frame processing result of the preprocessed ambient sound signal, extract the features of each audio segment, and use the features of each audio segment to form a segment-level feature representation sequence of the preprocessed ambient sound signal; Step 4: post-processing the segment-level feature representation sequence of the pre-processed ambient sound signal to obtain a final segment-level feature representation sequence of the pre-processed ambient sound signal; The post-processing includes a first post-processing, a second post-processing and a third post-processing; Step 5: learning a complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the preprocessed ambient sound signal, and the obtained complete semantic representation subspace is the complete temporal feature; The specific process of step five is as follows: The objective function of learning a complete semantic representation subspace under temporal constraints is expressed as: s.t.,(D k ) T a n <(D k ) T a m Where A is the final segment-level feature representation sequence of the ambient sound signal after preprocessing, D∈R d×p is the basis matrix of a subspace of dimension p, where p is the pre-specified subspace dimension, I p×p is the identity matrix of dimension p×p, D T is the transposed matrix of D, D k is the kth column vector in the basis matrix D, a n with a m are the features of the nth and mth audio segments in A respectively, and n <m,‖·‖ F represents the F-norm; The objective function of formula (3) is equivalently converted to: Among them, S(d,p) is a Stiefel manifold composed of all matrices of dimension d×p that satisfy the column orthogonality relation; Then calculate the manifold S(d,p) so that the objective function The matrix D with the smallest value is taken as the complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the preprocessed ambient sound signal.
2. A complete temporal feature extraction method for environmental sound recognition according to claim 1, characterized in that: The post-processing of the segment-level feature representation sequence of the pre-processed ambient sound signal is specifically performed as follows: The segment-level feature representation sequence of the preprocessed ambient sound signal is recorded as yes The feature representation of the nth audio segment in , n = 1, 2, ..., N, N is the total number of audio segments corresponding to the ambient sound signal after preprocessing, d is The dimension of , R is a real number, Indicated by The sequence of components; right Perform a first post-processing to obtain a first post-processing result; perform a second post-processing on the first post-processing result to obtain a second post-processing result; perform a third post-processing on the second post-processing result, and use the third post-processing result as the final segment-level feature representation sequence of the pre-processed ambient sound signal.
3. A complete temporal feature extraction method for environmental sound recognition according to claim 2, characterized in that: The specific process of the first post-processing is: in, It is the feature representation of the nth audio segment in the first post-processing result.
4. A complete temporal feature extraction method for environmental sound recognition according to claim 3, characterized in that: The specific process of the second post-processing is: in, is the feature representation of the nth audio segment in the second post-processing result, and ‖·‖ represents the 2-norm.
5. A complete temporal feature extraction method for environmental sound recognition according to claim 4, characterized in that: The specific process of the third post-processing is: Among them, |·| represents taking the absolute value, sign(·) represents the sign function, and a n is the feature representation of the nth audio segment in the final segment-level feature representation sequence; The final segment-level feature representation sequence of the preprocessed ambient sound signal is recorded as A = [a n ],[a n ] indicates that a 1 、a 2 , …, a N The sequence composed of.
6. A complete temporal feature extraction method for environmental sound recognition according to claim 5, characterized in that: The complete semantic representation subspace is used to identify the acquired environmental sound signal, and the specific method of identification is: The complete semantic representation subspace is used as the input of a support vector machine, and the support vector machine outputs the recognition result of the acquired environmental sound signal.
7. A complete temporal feature extraction method for environmental sound recognition according to claim 6, characterized in that: The training method of the support vector machine is: Step 1: Obtain an environmental sound signal training data set, and preprocess each environmental sound signal in the training data set to obtain each preprocessed environmental sound signal; Step 2: for the preprocessed i-th ambient sound signal in the training data set, perform frame processing on the preprocessed i-th ambient sound signal, and then perform feature extraction on each frame obtained by the frame processing, and use the extracted features of each frame to form a frame-level feature representation sequence of the preprocessed i-th ambient sound signal; Where i = 1, 2, ..., I, I is the total number of environmental sound signals in the training data set; Step 3: Segment the frame processing result of the i-th ambient sound signal after preprocessing in the training data set, extract the features of each audio segment respectively, and use the features of each audio segment to form a segment-level feature representation sequence of the i-th ambient sound signal after preprocessing; Step 4: post-processing the segment-level feature representation sequence of the i-th ambient sound signal to obtain a final segment-level feature representation sequence of the i-th ambient sound signal; Step 5: learning a complete semantic representation subspace corresponding to the final segment-level feature representation sequence of the i-th ambient sound signal; Step 6: Use the category labels of each signal in the environmental sound signal training data set and the learned complete semantic representation subspace to train the support vector machine.
Citation Information
Patent Citations
Acoustic event recognition method based on subspace representation learning
CN110148428A
Environment sound identification method and device, and readable storage medium
CN110827854A
Acoustic event identification method based on common subspace representation learning
CN113361592A
Video text retrieval method, system and equipment based on multi-semantic space and medium
CN116628261A
Target behavior recognition method and apparatus based on visual-audio feature fusion, and application
WO2023216609A1
