Characterization method of Erhu musical instrument timbre based on feature fusion

Abstract

The Erhu musical instrument timbre characterization method based on feature fusion comprises the following steps: S1, preprocessing collected original Erhu audios; s2, selecting a zero-crossing rate for the time domain feature extraction method of the preprocessed audio, reflecting the frequency characteristics of the audio signal, and capturing the instantaneous change of the signal; s3, selecting constant Q transformation for a frequency domain feature extraction method of the preprocessed audio; s4, selecting MFCCs as an inverted frequency domain feature extraction method for preprocessing the audio signal, and simulating human auditory characteristics to extract auditory characteristics of the audio; and S5, carrying out dimension removal and standardization processing on the three features extracted in the steps S3, S4 and S5, carrying out feature fusion by adopting feature splicing and early fusion, and generating a fusion feature vector representing the tone of the Erhu. The Erhu musical instrument timbre characterization method based on feature fusion can more comprehensively characterize the timbre characteristics of the Erhu musical instrument.

Description

Technical Field

[0001] The invention belongs to the technical field of national musical instrument recognition, and in particular relates to a method for characterizing the timbre of an erhu musical instrument based on feature fusion. Background Art

[0002] Audio features refer to features that contain representative information in audio signals. Most of these features originated from speech signal research at the end of the last century, and were subsequently extended to music signals and environmental sound signals with the development of artificial intelligence and deep learning technology. Feature extraction refers to the process of quantifying and extracting representative and important information from audio signals, and is also a pre-operation for establishing deep learning models. In computer science, sound is considered to be a series of digital signals generated by vibrations, and the audio feature extraction method analyzes and processes these digital signals to extract representative and distinguishable feature parameters. Timbre is an important audio feature, which refers to the different harmonic structures of sound waves when different instruments make sounds due to different materials and structures, thus producing a unique sound that is different from other sounds. This feature enables people to distinguish different instruments, and even when they play the same pitch and rhythm, they can be clearly distinguished by the human auditory system.

[0003] As one of the classic folk instruments, the erhu can be used as a typical object for musical instrument research. The current research method is to establish a standard database for multiple instruments and use deep learning technology to establish a timbre data model to achieve timbre recognition, but it can only classify different types of instruments and cannot complete the characterization of a single instrument such as material, category, playing technique or even emotional expression.

[0004] The timbre of the erhu has rich and varied characteristics, including basic timbre and emotional timbre. These timbre characteristics are not only related to the physical properties of the instrument (such as string material, sound box structure, etc.), but also closely related to the performance skills and emotional expression of the performer. The traditional method uses a single feature to characterize the timbre, which can only describe the timbre characteristics of erhu music from a specific aspect, and may not be able to capture all the timbre information of the audio signal; on the other hand, in the case of poor recording environment or unstable performance skills, a single feature may not accurately reflect the true timbre of the erhu instrument. These problems will affect the accurate characterization of the timbre of the erhu instrument. The erhu includes many categories such as cylindrical erhu, octagonal erhu, hexagonal erhu, and front octagonal and back round erhu. The timbre contained in different categories is different, and the current research methods can only achieve the recognition of the erhu as a large instrument classification. Due to the large number of erhu instruments and the rich and diverse emotional expressions and artistic conceptions of different categories, how to achieve the characterization of the timbre of the erhu through more advanced methods in the face of these situations is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a method for characterizing the timbre of an erhu instrument based on feature fusion in response to the problems in the prior art.

[0006] To this end, the above-mentioned purpose of the present invention is achieved through the following technical solutions: A method for characterizing the timbre of an erhu instrument based on feature fusion, characterized in that: S1, pre-processing the collected original erhu audio, including denoising, eliminating DC components, framing operations and windowing processing steps, to optimize and improve the audio quality for subsequent processing; S2, selecting the zero-crossing rate for the time domain feature extraction method of the pre-processed audio, reflecting the frequency characteristics of the audio signal and capturing the instantaneous changes of the signal; S3, selects constant Q transform as the frequency domain feature extraction method for the pre-processed audio, provides consistent frequency resolution, synchronously provides time domain and frequency domain information, has the ability to capture changes in pitch and timbre, and accurately identifies notes; S4, select MFCCs as the inverse frequency domain feature extraction method for preprocessing audio signals, and simulate human auditory characteristics to extract auditory features of audio; S5, de-dimensionalizing and standardizing the three features extracted in steps S3, S4, and S5, and fusing the features using feature concatenation and early fusion to generate a fused feature vector representing the timbre of the erhu.

[0007] While adopting the above technical solutions, the present invention may also adopt or combine the following technical solutions: As a preferred technical solution of the present invention: Step S1 comprises the following steps: S1.1: De-noise and eliminate DC component of music signal x(n) of Erhu instrument; S1.2: Perform a frame operation, the original music signal x(n) of the erhu instrument is framed, each short period is called a frame, the length of a frame is 5.50ms, and the audio signal of a frame is judged to be a stable signal for subsequent signal analysis and processing; S1.3: Perform Hamming windowing operation to eliminate spectrum leakage caused by fast Fourier transform and better reflect the nature of music, completing the preprocessing operation.

[0008] As a preferred technical solution of the present invention: Step S2 comprises the following steps: S2.1: Calculate the zero-crossing rate and set the music signal of the erhu instrument that has completed the preprocessing operation to x n (m), for each frame of music signal, the calculation formula is: , where sgn[ ] is the sign function, defined as: sgn[x(n)]= , where x n (m-1) refers to the value of the previous sample point of the current frame music signal; S2.2: The short-time zero-crossing rate ZCR of each frame of the music signal is obtained through step S2.1, and retained as the first extracted characteristic parameter representing the timbre of the erhu instrument.

[0009] As a preferred technical solution of the present invention: Step S3 comprises the following steps: S3.1: Let the original music signal of the Erhu instrument be x(n), then its CQT transform is defined as: ,in, represents the constant Q transform coefficient of the kth frequency bin, N is the normalization factor, and is the length of the window function; Refers to the window function, which is used to reduce spectrum leakage. The number of sampling points corresponding to the current frequency bin; Complex exponential basis functions, used for Fourier transform, is the window function after adding the Hamming window of length N; Q is the Q constant factor in the constant Q transform; k is the sampling point number of the CQT spectrum, and The length value is related to S3.2: Define the Q constant factor: , b is the number of frequency spectrum lines contained in one octave of erhu music; S3.3: Define the Kth frequency point number as: K=b[ ],[] means rounding in the positive direction. =4186Hz; S3.4: Definition Expressed as the center frequency of the kth subframe: ; S3.5: Save the CQT feature vector in the matrix In the example, the second characteristic parameter representing the timbre of the erhu instrument is retained.

[0010] As a preferred technical solution of the present invention: step S4 comprises the following steps: S4.1: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x w (i,n), frame length l = 1024, frame shift inc = 512, for x w (i,n) performs fast Fourier transform FFT, which is defined as: X(i,k)=FFT[x i (m)]; S4.2: Calculate the spectral line energy of each frame for X(i,k), that is, take the modulus and square it, which is defined as: E(i,k)=|X(i,k)| 2 ; S4.3: Calculate the energy of E(i,k) after passing through a set of Mel filters, defined as: , where the transfer function of each filter is defined as: , , where M is the number of filters; f(m) is the center frequency of each filter; i is the i-th frame; k is the k-th spectral line in the spectrum; S4.4: Logarithmize the energy passing through the Mel filter, defined as: s(i,m)=log[S(i,m)]; S4.5: Perform discrete cosine transform (DCT) on the logarithmic energy, defined as: MFCC(i,n)= , MFCC(i,n) is the i Frame No. n MFCC coefficients; S4.6: Save the MFCC feature vector in the matrix MFCC(i,n) and retain it as the third feature parameter extracted to represent the timbre of the erhu instrument.

[0011] As a preferred technical solution of the present invention: step S5 comprises the following steps: S5.1: Take the dimensionality reduction method of finding the average value to reduce the CQT sound matrix Each dimension is averaged; each dimension of the MFCC matrix MFCC(i,n) is averaged; S5.2: De-dimension and eliminate the order of magnitude of the processed feature vector, and use the deviation standard method for processing: ,in To standardize the data, and Respectively represent the minimum and maximum values ​​in the eigenvector, where x is the value of the original data point; S5.3: After standardization, direct early fusion, i.e., feature-level fusion, is performed by concatenating multiple feature vectors into a fused feature in a serial concat manner, which is defined as: , where a, b, and c are the dimensions of the three eigenvectors respectively.

[0012] Compared with the prior art, the method for characterizing the timbre of an erhu musical instrument based on feature fusion of the present invention has the following beneficial effects: by multimodally combining the three features of zero-crossing rate, CQT and MFCC, more comprehensive and detailed audio feature information is provided for the timbre characterization of the erhu: the fusion of three different types of features solves the limitation problem of the existing single-feature timbre characterization of the erhu, wherein the zero-crossing rate can use its own detection ability of the instantaneous change of the audio signal to supplement the insensitivity of CQT to dynamic changes, and solves the problem that the existing features cannot simultaneously capture the frequency structure and dynamic changes of the audio; CQT uses a higher frequency resolution than MFCC to improve the ability to process the pitch change of the musical instrument, while retaining the strong anti-noise ability of MFCC; and MFCC not only effectively extracts the auditory features of the audio, but also uses the fused zero-crossing rate and the robustness of CQT to overcome the problem that the current single MFCC feature is prone to lose part of the audio information, which improves the noise resistance of the system as a whole and can provide more comprehensive and detailed erhu audio feature information. The extracted features are connected in series in a certain order, and early fusion is used to form a fused feature vector, which solves the problem that the existing sound feature fusion lacks specificity for the sound of a certain instrument. This comprehensive feature fusion enables the model to more accurately characterize the complexity and uniqueness of the erhu timbre, especially capturing the subtle timbre changes in the erhu instrument, which serves as a more complete feature data input for subsequent scientific research on erhu music, such as deep learning classification and recognition algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The present invention is a flow chart of a method for characterizing the timbre of an erhu instrument based on feature fusion. DETAILED DESCRIPTION

[0014] The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

[0015] The purpose of the present invention is to propose a method for characterizing the timbre of an erhu instrument based on feature fusion. Specifically, the audio features of the erhu instrument in three different domains are extracted, standardized and then fused to become fused feature parameters that can characterize the timbre of the erhu instrument, which can be used as input data for subsequent scientific research and production applications in the field of erhu music.

[0016] For the original erhu music signal, the timbre characteristics of the instrument can be represented by the audio features of multiple music signals; for the music signal of the erhu instrument, the Zero Crossing Rate (ZCR) is selected as the feature extraction method in its time domain; the Constant Q transform (CQT) is selected as the extraction method in its frequency domain; the Mel Frequency Cepstral Coefficients (MFCC) is selected as the extraction method in its inverse frequency domain; the three extracted features are standardized and then feature fused so that they can become combined feature parameters that characterize the timbre of the instrument for use in the subsequent computer science research process.

[0017] The technical solution adopted by the method for extracting the timbre features of the erhu musical instrument based on feature combination disclosed in the present invention is: A method for extracting timbre features of an erhu musical instrument based on feature combination comprises the following steps: S1: Before extracting the audio features of the erhu music, the audio signal is preprocessed, including preprocessing, framing, windowing and other operations. The preprocessing process includes operations such as denoising and eliminating DC components; the original music signal x(n) of the erhu is a non-steady-state signal and needs to be framed. Each short period is called a frame, and the length of a frame is generally 5.50ms; after the framing operation, the windowing operation is performed to eliminate the spectrum leakage caused by the fast Fourier transform. The window functions such as the Hamming window and the Hanning window can be selected. For the music of the erhu, the Hamming window has a smoother spectrum feature. The Hamming window is selected for windowing to complete the preprocessing operation.

[0018] S2: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x n (m), according to the definition of zero-crossing rate, for each frame of music signal, the calculation formula is: , where sgn[] is the sign function, defined as: sgn[x(n)] = ; The short-time zero-crossing rate of each frame of the music signal is obtained by calculation , retained as the first extracted feature parameter representing the timbre of the erhu instrument.

[0019] S3: Let the original music signal of the Erhu instrument be x(n), then its CQT transformation is defined as: ,in is the window function after adding the Hamming window of length N; Q is the constant factor in the constant Q transform; k is the sampling point number of the CQT spectrum, and The length value is related to the Q constant factor: , b is the number of frequency lines contained in one octave of erhu music; the Kth frequency point number is defined as: K=b[ ],[] means rounding in the positive direction; definition Expressed as the center frequency of the kth subframe: ; Save CQT features in the matrix In the example, the second characteristic parameter representing the timbre of the erhu instrument is retained.

[0020] S4: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x w (i,n), frame length l = 1024, frame shift inc = 512; for x w (i,n) is subjected to fast Fourier transform; the spectral line energy of each frame of X(i,k) is calculated; the energy of E(i,k) after passing through a group of Mel filters is calculated; the energy passing through the Mel filters is logarithmized; the logarithmic energy is subjected to discrete cosine transform (DCT); the MFCC feature extraction is completed, and the third feature parameter representing the timbre of the erhu instrument is retained as the extracted one.

[0021] S5: Before the fusion of the three original feature parameters extracted, the CQT features and MFCC features need to be reduced in dimension, and the dimensionality reduction method of finding the average value is adopted; the processed feature vectors are de-dimensionalized and eliminated in order of magnitude, and the deviation standard method is adopted for processing; after standardization, direct early fusion, i.e., feature-level fusion, is performed; feature-level fusion can make up for the differences between the three features without significantly increasing the dimensionality, and better improve the degree of representation of the timbre of the erhu instrument.

[0022] As a preferred solution, the specific steps of S1 include: S1.1: De-noising and eliminating DC components of the music signal x(n) of the erhu instrument; S1.2: Perform a frame operation. The original music signal x(n) of the erhu instrument is a non-steady-state signal. For a non-steady-state signal, a frame operation needs to be performed. Each short period is called a frame. The length of a frame is generally 5.50ms. It is generally believed that the audio signal of a frame is a stable signal, so it can be used for subsequent signal analysis and processing; S1.3: Perform windowing operation. For the music of Erhu instrument, the Hamming window has a smoother spectrum feature, which can eliminate the spectrum leakage caused by fast Fourier transform and better reflect the nature of music. The Hamming window is selected for windowing to complete the preprocessing operation.

[0023] As a preferred solution, the specific steps of S2 include: S2.1: Calculate the zero-crossing rate and set the music signal of the erhu instrument that has completed preprocessing operations such as windowing and framing to xn (m), for each frame of music signal, the calculation formula is: , where sgn[] is the sign function, defined as: sgn[x(n)] = ; S2.2: Through the above calculation, the short-time zero-crossing rate ZCR of each frame of the music signal is obtained and retained as the first extracted characteristic parameter representing the timbre of the erhu instrument.

[0024] As a preferred solution, the specific steps of S3 include: S3.1: Let the original music signal of the Erhu instrument be x(n), then its CQT transform is defined as: ,in is the window function after adding the Hamming window of length N; Q is the Q constant factor in the constant Q transform; k is the sampling point number of the CQT spectrum, and The length value is related to S3.2: Define the Q constant factor: , b is the number of frequency spectrum lines contained in one octave of erhu music; S3.3: Define the Kth frequency point number as: K=b[ ],[] means rounding in the positive direction. =4186Hz; S3.4: Definition Expressed as the center frequency of the kth subframe: .

[0025] In feature extraction, b=12 is taken. =27.5Hz; S3.5: Save the CQT feature vector in the matrix In the example, the second characteristic parameter representing the timbre of the erhu instrument is retained.

[0026] As a preferred solution, the specific steps of S4 include: S4.1: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x w (i,n), frame length l = 1024, frame shift inc = 512. w (i,n) performs a fast Fourier transform (FFT), defined as: X(i,k)=FFT[x i (m)]; S4.2: Calculate the spectral line energy of each frame for X(i,k), that is, take the modulus and square it, which is defined as: E(i,k)=|X(i,k)| 2 ; S4.3: Calculate the energy of E(i,k) after passing through a set of Mel filters, defined as: , where the transfer function of each filter is defined as: , , where M is the number of filters; f(m) is the center frequency of each filter; i is the i-th frame; k is the k-th spectral line in the spectrum; S4.4: Logarithmize the energy passing through the Mel filter, defined as: s(i,m)=log[S(i,m)]; S4.5: Perform discrete cosine transform (DCT) on the logarithmic energy, defined as: MFCC(i,n)= ; S4.6: Save the MFCC feature vector in the matrix MFCC(i,n) and retain it as the third feature parameter extracted to represent the timbre of the erhu instrument.

[0027] As a preferred solution, the specific steps of S5 include: S5.1: Take the dimensionality reduction method of finding the average value to reduce the CQT sound matrix Each dimension is averaged; each dimension of the MFCC matrix MFCC(i,n) is averaged; S5.2: De-dimension and eliminate the order of magnitude of the processed feature vector, and use the deviation standard method for processing: ,in To standardize the data, and Respectively represent the minimum and maximum values ​​in the eigenvector; S5.3: After standardization, direct early fusion, i.e., feature-level fusion, is performed by directly concatenating multiple feature vectors into a fused feature in a serial connection (concat) manner, which is defined as: , where a, b, and c are the dimensions of the three eigenvectors respectively.

[0028] The present invention provides a method for characterizing the timbre of an erhu musical instrument based on feature fusion. Three single features that can characterize the timbre of the erhu musical instrument are extracted, standardized, and then serially connected into fused features to obtain features containing more timbre information. The features can be used as more complete feature data input in subsequent scientific research on erhu music, such as deep learning classification, recognition and other algorithms.

[0029] Feature fusion combines the three features of zero-crossing rate, CQT and MFCC in a multimodal manner and utilizes their complementarity to solve the problem of insufficient robustness of a single feature in an external noise environment and when the timbre of the erhu changes. The zero-crossing rate can make up for the insensitivity of CQT to dynamic changes, while CQT provides a higher frequency resolution than MFCC, which is suitable for capturing pitch changes, and MFCC effectively extracts the auditory features of the audio, especially providing more stable feature information when the noise is large. Through this feature fusion, the model can achieve a more comprehensive understanding of the complexity and uniqueness of the timbre of the erhu, significantly improving the accuracy of classification and recognition, while enhancing the stability of the model in complex environments.

[0030] The idea of ​​multi-feature modal fusion in the traditional field of image machine learning is introduced into the field of audio processing, which improves the one-sidedness of previous music signal processing that only extracts a single feature, and can more comprehensively characterize the timbre characteristics of the erhu instrument. Example 1

[0031] Please refer to Figure 1 , a method for characterizing the timbre of an erhu instrument based on feature fusion, comprising the following steps: S1: Before extracting the audio features of the erhu instrument music, the audio signal is preprocessed to reduce the interference of some other factors and improve the efficiency of subsequent feature recognition, and to simplify the subsequent analysis process, including denoising, eliminating DC components, framing and windowing operations.

[0032] The specific operation steps of S1 include: S1.1: De-noising and eliminating DC components of the music signal x(n) of the erhu instrument; S1.2: Perform a frame operation. The original music signal x(n) of the erhu instrument has an overall audio feature parameter that changes over time and is a non-steady-state signal. Non-steady-state signals need to be short-termed, that is, framed. Each short period is called a frame. The length of a frame is generally 5.50ms. It is generally believed that a frame of audio signal is a stable signal, so it can be used for subsequent signal analysis and processing; S1.3: Perform windowing operation to eliminate the spectrum leakage caused by fast Fourier transform. You can choose window functions such as Hamming window and Hanning window. For the music of erhu instrument, Hamming window has smoother spectrum characteristics and better reflects the nature of music. Select Hamming window for windowing to complete the preprocessing operation.

[0033] S2: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x n (m), calculate its zero-crossing rate and retain it as the first characteristic parameter.

[0034] The specific steps of S2 include: S2.1: The zero-crossing rate can, to a certain extent, be a good measure of the smoothness of an instrument in a piece of music, which happens to be an important property that reflects the timbre of the erhu instrument. Set the music signal of the erhu instrument that has been preprocessed by windowing and framing as x n (m), according to the definition of zero-crossing rate, for each frame of music signal, the calculation formula is: , where sgn[] is the sign function, defined as: sgn[x(n)] = ; S2.2: Through the above calculation, the short-time zero-crossing rate ZCR of each frame of the music signal is obtained and retained as the first extracted characteristic parameter representing the timbre of the erhu instrument.

[0035] S3: The frequency distribution of Erhu music is 294Hz~2352Hz, which conforms to a typical harmonic structure. Constant Q transform (CQT) is a characteristic parameter for music signals with harmonic structure, which can well reflect the linear relationship of human hearing perception and has good timbre representativeness. According to the definition of CQT, the CQT feature is saved in the matrix , reserved as the second characteristic parameter.

[0036] The specific steps of S3 include: S3.1: Let the original music signal of the Erhu instrument be x(n), then its CQT transform is defined as: ,in is the window function after adding the Hamming window of length N; Q is the Q constant factor in the constant Q transform; k is the sampling point number of the CQT spectrum, and The length value is related to S3.2: Define the Q constant factor: , b is the number of frequency spectrum lines contained in one octave of erhu music; S3.3: Define the Kth frequency point number as: K=b[ ],[] means rounding in the positive direction. =4186Hz; S3.4: Definition Expressed as the center frequency of the kth subframe: .

[0037] In feature extraction, b=12 is taken. =27.5Hz; S3.5: Save the CQT feature vector in the matrix In the example, the second characteristic parameter representing the timbre of the erhu instrument is retained.

[0038] S4: For an audio signal, the conversion relationship between its Mel frequency and linear frequency is defined as: , represents the perceptual frequency in mel, and f represents the linear frequency in Hz. According to the MFCC definition, the MFCC extraction operation is completed and retained as the third feature parameter.

[0039] The specific steps of S4 are: S4.1: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x w (i,n), frame length l = 1024, frame shift inc = 512. w (i,n) performs a fast Fourier transform (FFT), defined as: X(i,k)=FFT[x i (m)]; S4.2: Calculate the spectral line energy of each frame for X(i,k), that is, take the modulus and square it, which is defined as: spectral line energy E(i,k)=|X(i,k)| 2 ; S4.3: Calculate the energy of the spectral line E(i,k) after passing through a set of Mel filters, defined as: , where the transfer function of each filter is defined as: , , where M is the number of filters; f(m) is the center frequency of each filter; i is the i-th frame; k is the k-th spectral line in the spectrum; S4.4: Logarithmize the energy passing through the Mel filter, defined as: s(i,m)=log[S(i,m)]; S4.5: Perform discrete cosine transform (DCT) on the logarithmic energy, defined as: MFCC(i,n)= ; S4.6: Save the MFCC feature vector in the matrix MFCC(i,n) and retain it as the third feature parameter extracted to represent the timbre of the erhu instrument.

[0040] S5: Before the three original feature parameters are fused, the CQT feature and MFCC feature need to be reduced in dimension to avoid increasing the consumption of computer resources and processing efficiency in the later stage. The dimensionality reduction method of averaging is adopted; the processed feature vector is de-dimensionalized and the order of magnitude is eliminated, and the deviation standard method is adopted for processing; after standardization, direct early fusion, i.e. feature-level fusion, is performed; feature-level fusion can make up for the differences between the three features without significantly increasing the dimension, and better improve the degree of representation of the timbre of the erhu instrument by the fusion feature.

[0041] The specific steps of S5 include: S5.1: Take the dimensionality reduction method of finding the average value to reduce the CQT sound matrix Each dimension is averaged; each dimension of the MFCC matrix MFCC(i,n) is averaged; S5.2: De-dimension and eliminate the order of magnitude of the processed eigenvectors, and process them using the deviation standard method: ,in To standardize the data, and Respectively represent the minimum and maximum values ​​in the eigenvector; S5.3: After standardization, direct early fusion, i.e., feature-level fusion, is performed by concatenating multiple feature vectors into a fused feature in a serial connection (Concat) manner, which is defined as: , where a, b, and c are the dimensions of the three eigenvectors respectively.

[0042] The fused features obtained by fusing three single features that can represent the timbre can better represent the timbre properties of the erhu instrument.

[0043] By fusing the above three features, we can simultaneously obtain time domain information (zero-crossing rate captures dynamic changes), frequency domain information (CQT provides high-resolution spectrum), and auditory perception features (MFCC simulates the auditory characteristics of the human ear), which solves the problem that the existing sound feature fusion lacks the targeting of a certain instrument's sound. This comprehensive feature fusion enables the model to more accurately characterize the complexity and uniqueness of the erhu timbre, especially to capture the subtle timbre changes in the erhu instrument.

[0044] The present invention provides a method for characterizing the timbre of an erhu musical instrument based on feature fusion. Three single features that can characterize the timbre of the erhu musical instrument are extracted, standardized, and then serially connected into fused features to obtain features containing more timbre information. The features can be used as more complete feature data input in subsequent scientific research on erhu music, such as deep learning classification, recognition and other algorithms.

[0045] The idea of ​​multi-feature modal fusion in the traditional field of image machine learning is introduced into the field of audio processing, which improves the one-sidedness of previous music signal processing that only extracts a single feature, and can more comprehensively characterize the timbre characteristics of the erhu instrument.

[0046] The above-mentioned specific implementation methods are used to explain the present invention and are only preferred embodiments of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A method for characterizing the timbre of an erhu instrument based on feature fusion, characterized in that: S1, pre-processing the collected original erhu audio, including denoising, eliminating DC components, framing operations and windowing processing steps, to optimize and improve the audio quality for subsequent processing; S2, selecting the zero-crossing rate for the time domain feature extraction method of the pre-processed audio, reflecting the frequency characteristics of the audio signal and capturing the instantaneous changes of the signal; S3, selects constant Q transform as the frequency domain feature extraction method for the pre-processed audio, provides consistent frequency resolution, synchronously provides time domain and frequency domain information, has the ability to capture changes in pitch and timbre, and accurately identifies notes; S4, select MFCCs as the inverse frequency domain feature extraction method for preprocessing audio signals, and simulate human auditory characteristics to extract auditory features of audio; S5, de-dimensionalizing and standardizing the three features extracted in steps S3, S4, and S5, and fusing the features using feature concatenation and early fusion to generate a fused feature vector representing the timbre of the erhu.

2. The method for characterizing the timbre of an erhu musical instrument based on feature fusion as claimed in claim 1, characterized in that: Step S1 includes the following steps: S1.1: De-noise and eliminate DC component of music signal x(n) of Erhu instrument; S1.2: Perform a frame operation, the original music signal x(n) of the erhu instrument is framed, each short period is called a frame, the length of a frame is 5.50ms, and the audio signal of a frame is judged to be a stable signal for subsequent signal analysis and processing; S1.3: Perform Hamming windowing operation to eliminate spectrum leakage caused by fast Fourier transform and better reflect the nature of music, completing the preprocessing operation.

3. The method for characterizing the timbre of an erhu musical instrument based on feature fusion as claimed in claim 1, characterized in that: Step S2 includes the following steps: S2.1: Calculate the zero-crossing rate and set the music signal of the erhu instrument that has completed the preprocessing operation to x n (m), for each frame of music signal, the calculation formula is: , where sgn[ ] is the sign function, defined as: sgn[x(n)]= , where x n (m-1) refers to the value of the previous sample point of the current frame music signal; S2.2: The short-time zero-crossing rate ZCR of each frame of the music signal is obtained through step S2.1, and retained as the first extracted characteristic parameter representing the timbre of the erhu instrument.

4. A method for characterizing the timbre of an erhu instrument based on feature fusion as claimed in claim 1, characterized in that : Step S3 includes the following steps: S3.1: Let the original music signal of the Erhu instrument be x(n), then its CQT transform is defined as: ,in, represents the constant Q transform coefficient of the kth frequency bin, N is the normalization factor, and is the length of the window function; Refers to the window function, which is used to reduce spectrum leakage. The number of sampling points corresponding to the current frequency bin; Complex exponential basis functions, used for Fourier transform, is the window function after adding the Hamming window of length N; Q is the Q constant factor in the constant Q transform; k is the sampling point number of the CQT spectrum, and The length value is related to S3.2: Define the Q constant factor: , b is the number of frequency spectrum lines contained in one octave of erhu music; S3.3: Define the Kth frequency point number as: K=b[ ],[] means rounding in the positive direction. =4186Hz; S3.4: Definition Expressed as the center frequency of the kth subframe: ; S3.5: Save the CQT feature vector in the matrix In the example, the second characteristic parameter representing the timbre of the erhu instrument is retained.

5. The method for characterizing the timbre of an erhu musical instrument based on feature fusion as claimed in claim 4, characterized in that: In feature extraction, b=12 is taken. =27.5Hz.

6. The method for characterizing the timbre of an erhu musical instrument based on feature fusion as claimed in claim 1, characterized in that: Step S4 includes the following steps: S4.1: Set the music signal of the Erhu instrument that has completed preprocessing operations such as windowing and framing to x w (i,n), frame length l = 1024, frame shift inc = 512, for x w (i,n) performs fast Fourier transform FFT, which is defined as: X(i,k)=FFT[x i (m)]; S4.2: Calculate the spectral line energy of each frame for X(i,k), that is, take the modulus and square it, which is defined as: E(i,k)=|X(i,k)| 2 ; S4.3: Calculate the energy of E(i,k) after passing through a set of Mel filters, defined as: , where the transfer function of each filter is defined as: , , where M is the number of filters; f(m) is the center frequency of each filter; i is the i-th frame; k is the k-th spectral line in the spectrum; S4.4: Logarithmize the energy passing through the Mel filter, defined as: s(i,m)=log[S(i,m)]; S4.5: Perform discrete cosine transform (DCT) on the logarithmic energy, defined as: MFCC(i,n)= , MFCC(i,n) is the i Frame No. n MFCC coefficients; S4.6: Save the MFCC feature vector in the matrix MFCC(i,n) and retain it as the third feature parameter extracted to represent the timbre of the erhu instrument.

7. The method for characterizing the timbre of an erhu musical instrument based on feature fusion as claimed in claim 1, characterized in that: Step S5 includes the following steps: S5.1: Take the dimensionality reduction method of finding the average value to reduce the CQT sound matrix Each dimension is averaged; each dimension of the MFCC matrix MFCC(i,n) is averaged; S5.2: De-dimension and eliminate the order of magnitude of the processed eigenvectors, and process them using the deviation standard method: ,in To standardize the data, and Respectively represent the minimum and maximum values ​​in the eigenvector, where x is the value of the original data point; S5.3: After standardization, direct early fusion, i.e., feature-level fusion, is performed by concatenating multiple feature vectors into a fused feature in a serial concat manner, which is defined as: , where a, b, and c are the dimensions of the three eigenvectors respectively.

Images