High-resolution audio and EEG signal time-frequency analysis method based on frequency mapping

Abstract

The present invention discloses a high-resolution audio and EEG signal time-frequency analysis method based on a frequency mapping method, which relates to the technical field of signal time-frequency analysis, and includes the following steps: collecting audio and EEG signals, and preprocessing the input audio and EEG signals; segmenting the time series signal through a sliding time window, and preparing the signal segment in each time window for Fourier transform; and performing Fourier transform on each short time window signal. The present invention can effectively avoid the distortion of signal amplitude and the reduction of signal-to-noise ratio while improving the frequency resolution through the selection of precise phase adjustment amount and mapping mode function. For audio signals, it means that the processed signal can maintain the original timbre characteristics and auditory fluency; for EEG signals, it means that it can more accurately reflect the physiological activity state of the nervous system, providing reliable guarantee for the further analysis and application of EEG signals.

Description

Technical Field

[0001] The present invention relates to the technical field of signal time-frequency analysis, and in particular to a high-resolution audio and electroencephalogram signal time-frequency analysis method based on a frequency mapping method. Background Art

[0002] Time domain signals (i.e., time series signals) are signals whose amplitude varies with time, and they are widely present in nature, human life and production, and animal activities. These time domain signals carry a large amount of key characteristic information related to the signal generator, and are an important basis for understanding and studying the source object of the signal generator. Frequency is the basic parameter that describes the oscillation waveform signal, that is, the number of periodic changes of the waveform per unit time. The more periodic changes per unit time (the faster the signal changes), the higher the frequency; conversely, the lower the frequency. Time series signals can be expressed in the time domain as the change of signal amplitude relative to time, or in the frequency domain as the change of signal power (or other values) with frequency. As important time domain signals, the time-frequency feature analysis of audio signals and EEG signals is of great significance for signal processing and interpretation. However, traditional spectrum analysis cannot reflect the time-varying spectrum characteristics of non-stationary time series signals, while time-frequency analysis methods can provide the joint distribution of signals in time-frequency.

[0003] When providing the time-frequency joint distribution of the signal spectrum, the time-frequency analysis method faces the problem that the time resolution and frequency resolution cannot be taken into account at the same time. The shorter the sliding window length, the lower the frequency resolution, but the higher the time resolution; conversely, the longer the sliding window length, the higher the frequency resolution, but the lower the time resolution. This "uncertainty principle" makes it difficult for traditional time-frequency analysis methods to estimate the exact time-frequency representation of the signal. How to improve the frequency resolution while maintaining a high time resolution, so as to obtain a clearer time-frequency spectrum to accurately reflect the characteristic information of audio signals and EEG signals, is a technical problem that needs to be solved in the current field of time-frequency analysis. Summary of the invention

[0004] The purpose of the present invention is to provide a high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method includes the following steps:

[0007] Collect audio and EEG signals, and pre-process the input audio and EEG signals, including denoising and filtering, to improve the quality of the signals and provide a good basis for subsequent analysis;

[0008] The time series signal is divided by sliding time windows, and the signal segments in each time window are prepared for Fourier transform. The length of the time window should be selected according to the characteristics of the signal and the analysis requirements. For audio signals, the time window length may be shorter (such as 5ms), while for EEG signals, the time window length may be longer (such as 200ms or longer). By sliding the time window, the signal is divided into multiple short time window signals for subsequent spectrum analysis;

[0009] Perform Fourier transform on each short-time window signal, convert the time domain signal to the frequency domain, and obtain the signal spectrum, that is, the change of signal power (or other values) with frequency;

[0010] Design and apply mapping functions to map the spectrum to increase the distance between frequency components and improve frequency resolution. According to the mapping transformation mode function, calculate the phase adjustment amount and apply it to the mapped spectrum to maintain the relative amplitude relationship and signal-to-noise ratio of the signal to ensure the authenticity of the processing results.

[0011] The spectrum data after frequency mapping and phase adjustment is converted back to the time domain through inverse Fourier transform to obtain a signal with high time and frequency resolution. The inverse transform can obtain the signal after mapping in the frequency domain and then converting it back to the time domain, providing basic data for subsequent feature analysis and application;

[0012] Based on the time domain signal obtained by inverse Fourier transform, the time-frequency spectrum is redrawn, that is, the distribution of the signal at each time point and frequency point. By analyzing the time-frequency spectrum, the time-varying spectrum feature information of the signal is obtained, providing basic data for subsequent signal processing, feature extraction and pattern recognition.

[0013] A further improvement of the technical solution of the present invention is that the acquisition and preprocessing process of the audio and EEG signals is as follows:

[0014] Use special equipment and sensors to collect audio signals, convert them into digital format for computer processing, and perform pre-processing operations such as denoising, filtering, and standardization on the audio signals. Filtering technology is used to remove environmental noise and unnecessary frequency components, and adaptive filtering technology is used to further reduce noise interference. According to analysis requirements, filters (such as low-pass, high-pass, and band-pass filters) are selected to limit the frequency band of the signal. For speech signals, frequency components within the audible range of the human ear (20Hz to 20kHz) are retained, and the amplitude range of the signal is adjusted to make it suitable for the input requirements of subsequent processing algorithms. Signal standardization is achieved through linear or nonlinear transformation;

[0015] Perform endpoint detection on the voice signal to determine the start and end points of the voice signal, remove the silent segments and reduce the amount of data for subsequent processing;

[0016] The EEG cap is used to collect EEG signals on the surface of the scalp, which are amplified by an amplifier, and the EEG signals are preprocessed by denoising, filtering, baseline correction, and artifact removal.

[0017] According to the experimental design and task requirements, the preprocessed EEG signals are divided into different time periods, marked and classified.

[0018] A further improvement of the technical solution of the present invention is that the segmentation and Fourier transform process of the time series signal is as follows:

[0019] Determine the time window length based on the signal characteristics and analysis requirements , the time window length is in the number of samples;

[0020] For audio signals, the time window length is the number of samples corresponding to 5ms, and for EEG signals, the time window length is the number of samples corresponding to 200ms;

[0021] The signal is divided into multiple short time window signals through a sliding time window, and each time window has a fixed step size. (less than ) moves to cover the entire signal;

[0022] Mark each short-time window signal as ,in is the sample index, from 0 to .

[0023] A further improvement of the technical solution of the present invention is that the process of acquiring the spectrum diagram of the signal is as follows:

[0024] For each short-time window signal Apply discrete Fourier transform to obtain its frequency domain, expressed as ;

[0025] Calculate the absolute value of the amplitude and power spectrum , analyze the energy distribution of the signal at different frequencies;

[0026] Using the formula Frequency Index Convert to actual frequency value ,in, is the sampling frequency of the signal;

[0027] By calculating each short-time window signal and , and its corresponding frequency Pairing, constructing the spectrogram of the signal;

[0028] Repeat the processing of the entire signal to obtain the energy distribution of the signal at different times and frequencies, which is the time-frequency spectrum of the signal.

[0029] A further improvement of the technical solution of the present invention is that: the calculation formula for obtaining the frequency domain by applying discrete Fourier transform is:

[0030] ;

[0031] in, It is The complex amplitude of the frequency component, It is The short-time window signal value of samples, is the time window length, is the frequency index, , Is an imaginary unit.

[0032] A further improvement of the technical solution of the present invention is that the process of calculating the phase adjustment amount and applying it to the mapped spectrum is:

[0033] Assume the time domain signal is , the frequency domain is , the frequency domain mapping is transformed into ; Assume that after mapping transformation The processed time domain signal is , the frequency domain is ,but:

[0034] ; (1)

[0035] definition For the mapping mode function, different mapping mode functions are designed according to different expected outputs to obtain different mapping transformation results;

[0036] Calculate the phase adjustment amount according to the mapping transformation mode function and apply the phase adjustment amount to the mapped spectrum In order to compensate for the phase change introduced in the mapping process, the final mapping result is obtained.

[0037] A further improvement of the technical solution of the present invention is that the acquisition process of the signal with high time and frequency resolution is:

[0038] Prepare frequency mapped and phase adjusted spectral data samples ;

[0039] After processing, the spectrum data samples Apply an inverse discrete Fourier transform to convert it from the frequency domain back to the time domain;

[0040] For the signals divided into multiple short-time windows, inverse discrete Fourier transform is performed on them, and all the short-time window signals after inverse discrete Fourier transform are merged to reconstruct the complete time domain signal;

[0041] Check the reconstructed signal samples Whether to maintain the characteristics of the original signal and whether to improve the time and frequency resolution, further apply window function processing to the reconstructed signal to reduce boundary effects and optimize signal quality.

[0042] A further improvement of the technical solution of the present invention is that: the processed spectrum data samples Applying the inverse discrete Fourier transform, the calculation formula is:

[0043] ;

[0044] in, are the reconstructed signal samples, is the processed spectrum data sample, is the number of transformation points, which is equal to the time window length, is the sample index in the time domain, Is an imaginary unit.

[0045] A further improvement of the technical solution of the present invention is that the drawing process of the time-frequency spectrum diagram is:

[0046] The time domain signal obtained by inverse discrete Fourier transform is analyzed again in time-frequency direction, the sliding window technology is used to segment the signal in the time direction, and the signal samples reconstructed in each time window are Perform Fourier transform to obtain the spectrum of each time window ,in Indicates the sequence number of the time window;

[0047] The spectrum of each time window is calculated by taking the square of the spectrum modulus The power spectral density of , where the square of the spectrum modulus is calculated as: ; Indicates that in the time window and frequency index The power spectral density at ;

[0048] The power spectrum density in each short time window is arranged in time order to form a time-spectrum diagram, which shows the distribution of the signal at each time point and frequency point;

[0049] Draw a time-frequency spectrum to visualize it. In the time-frequency spectrum, the frequency On the vertical axis, the time window On the horizontal axis, the power spectral density Indicated by color;

[0050] Observe the time-frequency spectrum diagram, analyze the changes of the signal at different times and frequencies, and through the time-frequency spectrum diagram, intuitively see the trend of the signal's spectral components changing over time, thereby obtaining the signal's time-varying spectral feature information.

[0051] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0052] 1. The present invention provides a high-resolution audio and EEG signal time-frequency analysis method based on the frequency mapping method. By accurately selecting the phase adjustment amount and the mapping mode function, it can effectively avoid the distortion of the signal amplitude and the reduction of the signal-to-noise ratio while improving the frequency resolution. For audio signals, it means that the processed signal can maintain the original timbre characteristics and auditory fluency; for EEG signals, it means that it can more accurately reflect the physiological activity state of the nervous system, providing a reliable guarantee for the further analysis and application of EEG signals.

[0053] 2. The present invention provides a high-resolution audio and EEG signal time-frequency analysis method based on the frequency mapping method. The single harmonic frequency in the original spectrum is mapped to a higher frequency position through a mapping function, which increases the number of harmonic cycles in the time window, effectively reduces the divergence of the harmonic signal projection pulse in the frequency domain, increases the spacing of the harmonic signal in the frequency domain, and significantly improves the frequency resolution. Compared with the traditional time-frequency analysis method, it can greatly enrich the time-varying spectrum feature information of the signal while maintaining the original time resolution. For audio signals, it means that the harmonic components of different audio signals can be obtained more accurately, thereby improving the recognition and analysis accuracy of audio; for EEG signals, it means that the time position of EEG occurrence in each frequency band can be located more accurately, providing more accurate basic data for EEG feature analysis and practical applications.

[0054] 3. The present invention provides a high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping. After the spectrum data after frequency mapping and phase adjustment is converted back to the time domain through inverse Fourier transform, a time-frequency spectrum diagram with higher resolution can be drawn. The time-frequency spectrum diagram not only clearly shows the distribution of the signal at each time point and frequency point, but also provides strong support for the feature analysis and subsequent application of the signal through rich spectrum detail information. In the field of audio signal processing, high-resolution time-frequency spectrum diagrams are helpful to identify and analyze subtle features of audio signals. In the field of EEG signal processing, they are helpful to accurately identify and analyze EEG activities of different rhythms. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0056] Figure 1 is a flow chart of the method of the present invention;

[0057] Figure 2 The present invention is a flow chart for obtaining a signal with high time and frequency resolution;

[0058] Figure 3 It is a schematic diagram of spectrum analysis and time-frequency analysis;

[0059] Figure 4 For audio signal, spectrogram and real-time spectrogram (analysis window length 5ms);

[0060] Figure 5 For EEG signals, spectrograms and real-time spectrograms (analysis window length 200ms);

[0061] Figure 6 Spectra and spectrograms obtained for different analysis window lengths of audio data;

[0062] Figure 7 The result of mapping the audio signal spectrum to 4 times the original frequency position;

[0063] Figure 8 It is a schematic diagram comparing the frequency shift mapping transformation and the frequency shift coordinate transformation of 2-D audio pulse data;

[0064] Fig. 9 Comparison of the time-frequency analysis effects of conventional audio signal method and frequency mapping method (analysis window length 5ms);

[0065] Fig.10 This is a comparison of the time-frequency analysis effects of conventional EEG signal methods and frequency mapping methods (analysis window length 200ms);

[0066] Fig.11 It is a sound file and a time domain waveform;

[0067] Fig.12 for Fig.11 Audio signal time-frequency spectrum and spectrogram (analysis window length 10ms);

[0068] Fig.13 Comparison between the WSOLA method, slow playback without changing pitch, and the frequency mapping method (analysis window length 5ms);

[0069] Fig.14The blue whale sound mapping transformation is played by increasing the speed by 3 octaves [c(f)=8f] without changing the tempo;

[0070] Fig.15 The high-frequency harmonic signal is obtained by mapping and transforming the audio of a single person's speech;

[0071] Fig.16 for Fig.15 A timbre scheme diagram of the mapping result combination;

[0072] Fig.17 This is a comparison chart of the non-pitch slow playback results of polyphonic and harmonious music audio data using the WSOLA method and the frequency mapping method;

[0073] Fig.18 A frequency band diagram for rhythmic EEG activity divided by frequency;

[0074] Fig.19 It is the time-frequency diagram and spectrum diagram of EEG signal (analysis window length is 1000ms);

[0075] Fig. 20 This is the application diagram of mapping transformation on spontaneous EEG signals collected with eyes closed (amplitude spectrum, analysis time window 200ms);

[0076] Fig.21 The application of mapping transformation on spontaneous EEG signals collected with eyes closed (power spectrum);

[0077] Fig. 22 The application of mapping transformation on spontaneous EEG signals (power spectrum) is shown;

[0078] Fig.23 The application of mapping transformation on visual evoked EEG signals (VEP) (power spectrum) diagram;

[0079] Fig.24 The application (power spectrum) diagram of mapping transformation on laser evoked EEG signals (LEP);

[0080] Fig.25 The application of mapping transformation on EP42 EEG signal (power spectrum);

[0081] Fig.26 A diagram showing the application of mapping transformation when EEG signals are played as audio. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Embodiment 1, as Figure 1 As shown, the present invention provides a high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method, comprising the following steps:

[0084] Collect audio and EEG signals, and pre-process the input audio and EEG signals, including denoising and filtering, to improve the quality of the signals and provide a good foundation for subsequent analysis. Use special equipment and sensors to collect audio signals, convert them into digital format for computer processing, and perform denoising, filtering, and standardization pre-processing operations on the audio signals. Filtering technology is used to remove environmental noise and unnecessary frequency components, and adaptive filtering technology is used to further reduce noise interference. According to analysis requirements, filters (such as low-pass, high-pass, and band-pass filters) are selected to limit the frequency band of the signal. For speech signals, frequency components within the audible range of the human ear (20Hz to 20kHz) are retained, and the amplitude range of the signal is adjusted to meet the input requirements of the subsequent processing algorithm. Signal standardization is achieved through linear or nonlinear transformation, and endpoint detection is performed on speech signals. Determine the start and end points of the speech signal, remove the silent segments and reduce the amount of data for subsequent processing, use an EEG cap to collect EEG signals on the scalp surface, amplify them through an amplifier, and perform preprocessing operations such as denoising, filtering, baseline correction, and artifact removal on the EEG signals. Among them, use independent component analysis (ICA) or principal component analysis (PCA) and other technologies to separate and remove noise, thereby removing bioelectric noise such as electromyographic interference and electrocardiographic interference. According to the frequency band characteristics of the EEG signal (δ, θ, α, β, γ waves), select the corresponding filter for band limiting, adjust the baseline level of the EEG signal to eliminate long-term drift and offset, use regression methods or adaptive filtering technology to identify and remove artifacts caused by blinking, head movement, etc., to reduce the impact of artifacts, and divide the preprocessed EEG signals into different time periods according to the experimental design and task requirements, and mark and classify them;

[0085] The time series signal is divided by sliding time windows, and the signal segments in each time window are prepared for Fourier transform. The length of the time window should be selected according to the characteristics of the signal and the analysis requirements. For audio signals, the time window length may be shorter (such as 5ms), while for EEG signals, the time window length may be longer (such as 200ms or longer). By sliding the time window, the signal is divided into multiple short time window signals for subsequent spectrum analysis. The time window length is determined according to the characteristics of the signal and the analysis requirements. The time window length is in the number of samples. For audio signals, the time window length is the number of samples corresponding to 5ms. For EEG signals, the time window length is the number of samples corresponding to 200ms. The signal is divided into multiple short time window signals through a sliding time window. Each time window is incremented by a fixed step size. (less than ) to cover the entire signal, and mark each short-time window signal as ,in is the sample index, from 0 to ;

[0086] Perform Fourier transform on each short-time window signal, convert the time domain signal to the frequency domain, and obtain the signal spectrum, that is, the change of signal power (or other values) with frequency. Apply discrete Fourier transform to obtain its frequency domain, expressed as , calculate the absolute value of the amplitude and power spectrum , analyze the energy distribution of the signal at different frequencies, using the formula Frequency Index Convert to actual frequency value ,in, is the sampling frequency of the signal, obtained by calculating each short-time window signal and , and its corresponding frequency Pairing, constructing the spectrum of the signal, repeatedly processing the entire signal, and obtaining the energy distribution of the signal at different times and frequencies, which is the time-frequency spectrum of the signal;

[0087] Furthermore, the calculation formula in the frequency domain is obtained by applying discrete Fourier transform:

[0088] ;

[0089] in, It is The complex amplitude of the frequency component, It is The short-time window signal value of samples, is the time window length, is the frequency index, , is an imaginary unit;

[0090] Design and apply a mapping function to map the spectrum to increase the distance between frequency components and improve frequency resolution. According to the mapping transformation mode function, calculate the phase adjustment amount and apply it to the mapped spectrum to ensure the authenticity of the processing results. The process of calculating the phase adjustment amount and applying it to the mapped spectrum is as follows:

[0091] Figure 7 The result of mapping the spectrum (amplitude spectrum) of an audio signal to 4 times the original frequency. Original spectrum Figure 7 After (a) is mapped to 4 times the frequency position Figure 7 In (b), the frequency resolution is significantly improved, the amplitude spectrum curve is sharper, and the spectrum details are richer and clearer.

[0092] Assume the time domain signal is , the frequency domain is , the frequency domain mapping is transformed into ; Assume that after mapping transformation The processed time domain signal is , the frequency domain is ,but:

[0093] ; (1)

[0094] definition For the mapping mode function, different mapping mode functions are designed according to different expected outputs to obtain different mapping transformation results. The mapping mode function formula is defined and designed as follows:

[0095] Octave Mapping: ; , is a positive constant (2)

[0096] Frequency Shift Mapping: ; is a constant (3)

[0097] Substituting the mapping functions of (2) and (3) into (1) in different orders, we can obtain different mapping results:

[0098] ; The spectrum is mapped to a times the original frequency in the frequency direction;

[0099] ; The spectrum is shifted in the frequency direction and mapped to the original frequency plus b;

[0100] or: ; After frequency doubling by a times, frequency shift mapping is performed by b;

[0101] ; After frequency shift mapping b, then frequency doubling mapping a times;

[0102] Analysis shows that for the same pattern function, different combination orders will give different results. ;

[0103] The phase adjustment amount is calculated according to the mapping transformation mode function. Taking the audio signal as an example, the calculation method of the phase adjustment amount is:

[0104] Define audio consonants as , It is a time signal with normalized amplitude, which has nothing to do with the reception time and amplitude of the audio signal, but the consonant completely contains all the auditory and spectral properties of a single audio signal at a certain moment. The consonant received at the time is recorded as , a single audio signal received at different times has different auditory and spectral properties;

[0105] At any time A single audio signal received is , whose amplitude is , It is expressed as:

[0106] ; (4)

[0107] Overall audio signal For all time( ) The sum of the received single audio signals:

[0108] ; (5)

[0109] The time direction of equation (5) is The Fourier transform of the audio signal is :

[0110] ; (6)

[0111] (6) In the formula, for The consonants received at all times The spectrum and the reception time Not relevant;

[0112] The frequency mapping transformation is essentially the spectrum of the consonant signal ( ) is used for mapping transformation, and the amplitude must be maintained (i.e.: ) and acceptance time (ie: )constant;

[0113] Assume the frequency mapping transformation mode function is (frequency mapping), (6) after frequency mapping transformation is:

[0114] ; (7)

[0115] (7) In the formula, It is the phase adjustment term of the mapping transformation, The reception time of the consonant after the mapping transformation must be met remain unchanged, so we have:

[0116]

[0117] get: ; (8)

[0118] Similarly, the frequency mapping transformation mode function can be obtained as follows: (frequency shift mapping), the mapping transformation phase adjustment amount is a constant that is independent of frequency:

[0119] ; (9)

[0120] The phase adjustment process of mapping transformation is the key to mapping transformation, and it is also the difference from coordinate transformation. Coordinate transformation only changes the frequency position of the spectrum function, but does not change the value of the spectrum function (frequency domain interpolation belongs to frequency coordinate transformation), while mapping transformation will change both the frequency position and function value of the spectrum function.

[0121] Apply the phase adjustment term of (8) or (9) to (7) to complete the frequency mapping transformation of the spectrum of time domain signals such as audio, and apply the phase adjustment amount to the mapped spectrum. In order to compensate for the phase change introduced in the mapping process, the final mapping result is obtained;

[0122] The spectrum data after frequency mapping and phase adjustment is converted back to the time domain through inverse Fourier transform to obtain a signal with high time and frequency resolution. The inverse transform can obtain the signal after mapping in the frequency domain and then converting it back to the time domain, providing basic data for subsequent feature analysis and application;

[0123] Based on the time domain signal obtained by inverse Fourier transform, the time-frequency spectrum is redrawn, that is, the distribution of the signal at each time point and frequency point. By analyzing the time-frequency spectrum, the time-varying spectrum feature information of the signal is obtained, providing basic data for subsequent signal processing, feature extraction and pattern recognition.

[0124] Embodiment 2, as Figure 2As shown, based on Example 1, the present invention provides a technical solution: Preferably, the acquisition process of the signal with high time and frequency resolution is:

[0125] Prepare frequency mapped and phase adjusted spectral data samples , for the processed spectrum data samples Apply inverse discrete Fourier transform to convert it from frequency domain back to time domain. For signals divided into multiple short-time windows, perform inverse discrete Fourier transform on them, merge all short-time window signals that have undergone inverse discrete Fourier transform, reconstruct the complete time domain signal, and check the reconstructed signal samples. Whether the characteristics of the original signal are maintained, and whether the time and frequency resolution are improved, and the reconstructed signal is further processed by applying a window function to reduce boundary effects and optimize signal quality;

[0126] Furthermore, the processed spectrum data samples Applying the inverse discrete Fourier transform, the calculation formula is:

[0127] ;

[0128] in, are the reconstructed signal samples, is the processed spectrum data sample, is the number of transformation points, which is equal to the time window length, is the sample index in the time domain, is an imaginary unit;

[0129] The process of drawing the time-frequency spectrum is as follows:

[0130] The time domain signal obtained by inverse discrete Fourier transform is analyzed again in time-frequency direction, the sliding window technology is used to segment the signal in the time direction, and the signal samples reconstructed in each time window are Perform Fourier transform to obtain the spectrum of each time window ,in Indicates the serial number of the time window. The spectrum of each time window is calculated by taking the square of the spectrum modulus. The power spectral density of , where the square of the spectrum modulus is calculated as: ; Indicates that in the time window and frequency index The power spectrum density at each short time window is arranged in time order to form a time-spectrum diagram. The time-spectrum diagram shows the distribution of the signal at each time point and frequency point. The time-spectrum diagram is drawn to visualize it. In the time-spectrum diagram, the frequency On the vertical axis, the time window On the horizontal axis, the power spectral density By using color representation and observing the time-frequency spectrum, we can analyze the changes of the signal at different times and frequencies. Through the time-frequency spectrum, we can intuitively see the trend of the spectral components of the signal changing over time, thereby obtaining the time-varying spectral characteristic information of the signal.

[0131] Spectrum Analysis

[0132] Spectral analysis (i.e., spectral estimation) is mathematically defined as estimating the spectral density of a random process from a series of time samples of the random process. Spectral estimation is performed by Fourier transforming the overall signal ( Figure 3 ), transforms the time domain signal into the frequency domain, and provides the power, amplitude, and phase change curves of the signal along the frequency (i.e., spectrum). Spectral analysis assumes that the signal spectrum does not change with time, and estimates it based on the overall signal. The result is the average effect of the overall signal, which does not contain any time information and cannot show the time-varying spectrum characteristics of the signal.

[0133] Time-frequency analysis

[0134] Non-stationary time series signals have the characteristics of statistical characteristics and spectral density that change rapidly over time, and their spectral time-varying characteristics convey important information about the signal. Spectral analysis cannot identify and reflect these important time-varying spectral characteristics of non-stationary time series signals, while time-frequency analysis methods can provide the joint time-frequency distribution of signal spectrum (power, amplitude, phase, etc.) in time-frequency.

[0135] The signal is segmented into sliding short-time windows in the time direction, and the spectrum analysis is performed on the short-time window signals at different time positions respectively ( Figure 3 ), we can get the time-frequency analysis results that reflect the time-frequency joint distribution characteristics of the signal (real-time spectrum: the distribution of the signal at each time point and frequency point). The time-frequency analysis results can well reflect the time-varying spectrum characteristics of non-stationary time series signals.

[0136] The important characteristics of time-frequency analysis are its frequency resolution and time resolution. There is a trade-off between frequency resolution and time resolution, namely the "uncertainty principle" of time-frequency analysis: increasing frequency resolution will reduce time resolution; and increasing time resolution will reduce frequency resolution. Figure 4 This is the time-frequency analysis result of a speech signal. When the short time window length is 5ms, the time resolution is good, but the frequency resolution is low.

[0137] Figure 5 This is the time-frequency analysis result of an EEG signal. When the short time window length is 200ms, the time resolution can be basically guaranteed, but the frequency resolution is low.

[0138] The "uncertainty principle" of time-frequency analysis is the fundamental limitation of the time-frequency analysis method in terms of time-frequency resolution. Overcoming this limitation is an effective way to improve the accuracy of time-frequency analysis.

[0139] A brief introduction to the "uncertainty principle" of time-frequency analysis

[0140] The dilemma of time-frequency analysis method: time resolution and frequency resolution cannot be taken into account at the same time;

[0141] When performing time-frequency analysis on non-stationary time series signals, the shorter the sliding window length, the lower the frequency resolution, but the higher the time resolution; conversely, the longer the sliding window length, the higher the frequency resolution, but the lower the time resolution. The "uncertainty principle" of time-frequency analysis makes it impossible to take into account both time resolution and frequency resolution at the same time, and it is impossible to estimate the exact time-frequency representation of the signal. That is: either it is accurate in frequency but not in time, or it is accurate in time but not in frequency.

[0142] Figure 6 The time-frequency spectrograms obtained for different analysis window lengths of audio data. Figure 6 (b) in the figure shows the audio data and analysis time window; Figure 6 (a) in the figure is the time-frequency spectrum (upper) and spectrum diagram (lower) obtained by long time window analysis. The long time window makes the frequency resolution higher, and the harmonics in the audio can be accurately separated (white circle position), but the time resolution becomes lower, and the harmonics cannot be distinguished in the time direction (black circle position); Figure 6 (c) in the figure is the time-frequency spectrum (top) and spectrum diagram (bottom) obtained by short-time window analysis. The short-time window makes the frequency resolution lower, and the harmonics in the audio cannot be separated (white circle position), but the time resolution is higher, and the harmonics can be better distinguished in the time direction (black circle position).

[0143] Increasing the spacing between frequency components (such as audio harmonics) in the frequency direction is to improve the time-frequency spectrum.

[0144] The key to frequency resolution

[0145] In summary, due to the limitation of the "uncertainty principle" of time-frequency analysis, time-frequency analysis cannot take into account both time resolution and frequency resolution. The purpose of the present invention is to increase the spacing of each frequency component (such as audio harmonics) in the frequency direction while maintaining the high time resolution of short-time window analysis, so as to achieve the purpose of improving the frequency resolution of the time-frequency spectrum.

[0146] High-resolution audio and EEG signal time-frequency analysis technology based on frequency mapping method

[0147] The purpose of high-resolution audio and EEG signal time-frequency analysis technology based on frequency mapping method

[0148] Time-spectrum analysis is the basis for many studies on the characteristic information of time-domain signals (such as audio signals and EEG signals). The time and frequency resolution of the time-spectrum analysis will directly affect the analysis accuracy of the characteristic information of time-domain signals.

[0149] In time-frequency analysis, since a short time window is used for analysis, although a high time resolution is guaranteed, the frequency resolution is reduced, so that the frequency components of the time series signal cannot be well separated in the time-frequency spectrum, the time-frequency spectrum details are blurred, and the accuracy of the time-frequency analysis is affected.

[0150] The purpose of high-resolution time-frequency analysis technology based on frequency mapping method is to propose a new time-frequency analysis method for time series signals to increase the distance between frequency components in the time-frequency spectrum, improve its frequency resolution, and thus improve the clarity and detail identifiability of the time-frequency spectrum.

[0151] The high-resolution time-frequency analysis technology based on the frequency mapping method hopes to improve the frequency resolution of the time-frequency spectrum while maintaining the relative amplitude relationship and signal-to-noise ratio of the time series signal as much as possible, thereby ensuring the authenticity of the processing results (for audio signals, ensuring the auditory smoothness and original timbre of the signal after processing by this technology is a concrete manifestation of the authenticity of the processing results).

[0152] The principle and implementation of time-frequency analysis technology based on frequency mapping method

[0153] Principle of time-frequency analysis technology using frequency mapping method

[0154] When using a sliding time window to split the signal for Fourier transform, the more cycles of a single harmonic signal contained in the time window, the sharper the projection pulse of the harmonic signal in the frequency domain (i.e., the spectrum amplitude value in the frequency domain) and the higher the frequency resolution. Conversely, the fewer cycles contained in the time window, the more divergent the projection pulse in the frequency domain, which is the fundamental reason for the low frequency resolution and the inability to distinguish adjacent harmonic signals. Increasing the spacing of harmonic signals in the frequency domain and reducing the divergence of the projection pulse of the harmonic signal in the frequency domain are the only way to solve the problem.

[0155] The basic principle of high-resolution time-frequency analysis technology based on frequency mapping method: design the corresponding mapping function, map the spectrum of a single harmonic at all frequency positions in the spectrum function to a higher frequency position, and raise the frequency of the harmonic signal. After frequency mapping, the frequency of the harmonic signal becomes higher, increasing the number of cycles of the harmonic in the time window, thereby reducing the divergence of the harmonic signal projected pulse in the frequency domain, increasing the spacing of the harmonic signal in the frequency domain, and achieving the purpose of improving the frequency resolution of the time-frequency spectrum.

[0156] Figure 8 For a 2-D audio pulse data Figure 8 (a) Frequency shift mapping transformation Figure 8 (b) and frequency shift coordinate transformation Figure 8 Comparison of the results in (c) in Figure 1. The frequency shift mapping transformation adjusts the phase of the audio data, and the phase is not distorted, ensuring the auditory fluency of the time domain audio signal. However, the phase of the frequency shift coordinate transformation result is distorted, which will deteriorate the auditory fluency of the audio signal.

[0157] Time domain representation of frequency mapping results

[0158] The frequency spectrum data of the audio and other time domain signals which have undergone the frequency mapping transformation are inversely Fourier transformed into the time domain to obtain a time domain signal with higher frequency and time resolution;

[0159] Analysis of the application effect of high-resolution audio and EEG signal time-frequency analysis technology based on frequency mapping method

[0160] Analysis of the effect of time-frequency analysis of audio signal frequency mapping method

[0161] Fig. 9 The figure is a comparison of the time-frequency analysis effects of the conventional method and the frequency mapping method for an audio signal (the analysis time window length is 5ms). Fig. 9 (a) is the audio signal, Fig. 9 (b) is the time-frequency spectrum of the conventional method time-frequency analysis. Fig. 9 (c) is the time-frequency analysis spectrogram of the frequency mapping method. The frequency resolution of the conventional method time-frequency analysis spectrogram is low, and the harmonics cannot be well distinguished. The frequency mapping method time-frequency analysis spectrogram maintains the original time resolution while greatly improving the frequency resolution. More harmonic frequency components are clearly visible, and the similarity between the high-resolution spectrogram and the original audio spectrogram is well maintained.

[0162] The frequency mapping method of time-frequency analysis improves the analysis accuracy of the time-frequency diagram of the audio signal, and provides better basic data for the characteristic analysis and subsequent applications of the audio signal.

[0163] Analysis of the effect of time-frequency analysis of EEG signal frequency mapping method

[0164] Fig.10 This is a comparison of the time-frequency analysis effects of the conventional method and the frequency mapping method for EEG signals (the analysis time window length is 200ms). Fig.10 (a) is the EEG signal. Fig.10 (b) is the time-frequency spectrum of the conventional method time-frequency analysis. Fig.10 (c) in the figure is the time-frequency analysis spectrogram of the frequency mapping method. The conventional method time-frequency analysis spectrogram has a very low frequency resolution, and the EEG components of different frequency bands cannot be distinguished. The frequency mapping method time-frequency analysis spectrogram greatly improves the frequency resolution, effectively separates the EEG signals of each frequency band, and can more accurately display the time duration range of the EEG signals of each frequency band.

[0165] The frequency mapping method provides a higher frequency and time resolution for time-frequency analysis, and is an effective tool for EEG signal analysis and application. EEG time-frequency analysis with high frequency and time resolution will certainly play an important role in EEG feature analysis and practical applications such as brain-computer interface technology for continuous and complex movements.

[0166] A brief description of the basic characteristics of audio signals and the application effects of frequency mapping transformation time-frequency analysis

[0167] A brief description of the basic characteristics of audio signals

[0168] Audio signals usually refer to sound wave signals that can be heard by the human ear. Infrasound waves that are inaudible to the human ear, ultrasound waves, and sound waves that can be heard by the human ear all belong to the same physical phenomenon and are still audio signals in a broad sense.

[0169] Audio signals are usually composed of multiple sound source signals, and are very complex signals. A speech signal produced by only one person is just a special case of an audio signal, that is, an audio signal with only one sound source, and its structure is relatively simple.

[0170] An audio signal is a periodic vibration signal that changes with time. Its meaning and content (auditory perception attributes) are perceived through the auditory system associated with time. Therefore, the meaning and content of an audio signal can only be "heard".

[0171] The visual system has no direct relationship with time, so it is difficult for us to directly "see" the meaning and content of the audio signal from the time domain waveform display of the audio signal. Fig.11 .

[0172] The auditory perception attributes perceived by the auditory system include sound attributes such as pitch, timbre and intonation. The auditory system is equivalent to performing a "Fourier transform" on the time domain audio signal, obtaining the spectrum of the audio signal and the information on how the spectrum changes over time (i.e., the time-spectrum graph), thereby perceiving attributes such as pitch, timbre and intonation.

[0173] The time-spectrum diagram of an audio signal is also called the spectrogram of the audio signal (called the spectrogram in speech analysis). The time-spectrum diagram more intuitively displays a lot of attribute information of the audio signal, so we can indirectly "see" a lot of audio attribute information from the time-spectrum diagram. According to the time-spectrum diagram, we can understand the sound at a speed much higher than the audio playback speed. Fig.12 for Fig.11 The time-frequency spectrum and spectrum diagram of the audio signal. The time-frequency spectrum diagram clearly shows the changes in the time-frequency characteristics of the audio signal, providing rich information for the analysis and research of the audio signal.

[0174] The time-frequency spectrum is a very effective tool for audio signal analysis and can be applied to fields such as intelligent speech, acoustic medicine, animal acoustics, sound source localization, and entertainment acoustics.

[0175] Comparison of audio signal frequency mapping method with other time-frequency analysis methods

[0176] The invariant pitch slow playback audio processing technology actually increases the frequency of the audio signal in proportion in the frequency direction, and its function is similar to frequency mapping. The method based on waveform similarity is the mainstream technology for invariant pitch slow playback audio processing (there are many implementation methods, and the most representative and mature method is the WSOLA algorithm: Waveform SimilarityOverLap-Add). In order to overcome problems such as poor auditory fluency (pitch break), the WSOLA method calculates the time shift of the repeated copies of the original audio signal based on the waveform similarity of the audio signal, and reconstructs the new audio signal based on the time shift.

[0177] Limitations of Waveform Similarity Method - Attenuates or even distorts weaker signals whose pitch is different from the loudest ones

[0178] Since audio signals are mostly multi-source signals, they contain sound signals of different pitches. When sound signals of different pitches sound at the same time, they overlap in the time domain and cannot be separated. Repeated copying and superposition will inevitably cause overlap or loss of signals in some time periods.

[0179] The waveform similarity calculation is only sensitive to a certain pitch signal with a strong volume, and can obtain a more accurate time shift suitable for the pitch signal, and better complete the reconstruction of the pitch signal. However, this time shift is not suitable for other signals with weaker volumes and pitches different from the strongest volume. After the signal is reconstructed, these weaker volume signals will inevitably be attenuated or even distorted, making the auditory fluency of the entire audio signal worse;

[0180] Frequency mapping method - overcomes the problem of attenuating or even distorting weak volume signals

[0181] The frequency mapping method maps sound signals of different pitches independently in the frequency domain, overcoming the problem of attenuating or even distorting weaker volume signals.

[0182] Fig.13This is a comparison between the results of the 1 / 4 slow playback without pitch change using the WSOLA method and the results of the frequency mapping method [c(f)=4f] (analysis window length 5ms). The harmonics of the distorted weak volume signal in the WSOLA processed audio are accurately reconstructed in the audio processed by the frequency mapping method, and the frequency resolution of the audio processed by the frequency mapping method is higher (black circle and black arrow position). When the audio file of the WSOLA method is played, there is an obvious stuttering phenomenon and the auditory fluency is deteriorated. However, when the audio file of the frequency mapping method is played at 1 / 4 slow speed, there is no stuttering or timbre distortion, and the timbre and fluency of the original audio file are completely maintained.

[0183] (3) Application effect of audio signal frequency mapping transformation time-frequency analysis

[0184] The frequency-mapped audio signal spectrum data is inversely Fourier transformed to the time domain to obtain a time domain signal with higher frequency and time resolution. Slow playback of the time domain audio signal can improve the time resolution of the audio signal, and more detailed information can be distinguished when listening. Combining the mapping transformation results of different mapping mode functions can obtain audio data with auditory effects.

[0185] Mapping transformation realizes the playback of blue whale sound with constant speed and raised by 3 octaves [c(f)=8f]

[0186] The frequency range of some animal sounds is beyond the human ear's perception range, and the human ear cannot hear them normally, which brings great difficulties to animal research. The sounds of large animals are often relatively low, and their pitch must be raised to be heard by the human ear.

[0187] Fig.14 This is an example of playing the blue whale sound mapping transformation without changing the tempo by raising it by 3 octaves [c(f)=8f].

[0188] Fig.14 (b) is the original audio played at normal speed: Since the original speech frequency is too low and is near the lower limit of human hearing, the human ear cannot perceive the audio information normally, making the originally smooth audio signal sound unsmooth. Since the original speech frequency is near the lower limit of human hearing, the unsmoothness is entirely caused by the hearing characteristics of the human ear and has nothing to do with the audio data.

[0189] Fig.14 (a) is played at 8 times the original speed: the human ear can hear normally, and the timbre and fluency are very good. However, due to the 8 times faster playback speed, the temporal resolution of the human ear is greatly reduced.

[0190] Fig.14(c) in the figure is a mapping transformation, which raises the speed by 3 octaves [c(f)=8f] and plays at normal speed without changing the speed: the mapping transformation maps all frequency components to a higher frequency position that is normally audible to the human ear (8 times the original frequency), thus increasing the spacing between harmonics and improving the frequency resolution.

[0191] After the mapping transformation, when the audio is played at normal speed, the human ear can hear it normally, and the timbre and fluency are very good. Because it is played at normal speed, the human ear's hearing resolution in time is maintained, and more sound details can be distinguished. The frequency and time resolution of the spectrogram have been improved.

[0192] Mapping transformation to achieve timbre and pitch changes of single-person speech

[0193] In order to obtain better and more novel auditory effects, human voices often need to be processed by speed change, pitch change, and timbre change. Some voice effect processing is highly dependent on rich high-frequency harmonic signals, and obtaining high-frequency harmonic signals has always been a difficult problem in voice processing. The higher the frequency resolution, the more conducive it is to the recovery of high-frequency harmonic signals.

[0194] The frequency mapping method provides an effective means to obtain high-quality higher-frequency harmonic components, ensuring the auditory smoothness of timbre and pitch changes.

[0195] Fig.15 The spectrogram shows the combination of single-person audio mapping transformation. By combining the mapping results of the mapping mode functions 2f and f+1pitch and the original audio data, audio data with higher harmonic components is obtained.

[0196] Raw audio data Fig.15 (a) is a single male voice counting from 1 to 12 in Mandarin. After the mapping function 2f is used to map all harmonics of the original audio to 2 times the frequency, audio data with higher frequency resolution is obtained. Fig.15 (b) in the above. Fig.15 The audio data in (b) is transformed by the mapping mode function f+1pitch mapping. Fig.15 (b) All harmonics of the audio are mapped to the frequency position of one higher fundamental tone Fig.15 (c) in .

[0197] The original audio and the results of the mapping mode function 2f and f+1 pitch mapping transformation are combined to obtain new audio data with the same pitch as the original audio but containing more high-frequency harmonic components (black circle position). Fig.15 (d) in the figure. Rich harmonic signals provide more basic data for processing speech timbre and pitch changes.

[0198] Fig.16 for Fig.15 An example of a timbre scheme for mapping result combination: the pitch of the voice remains unchanged (i.e. the fundamental frequency remains unchanged), but there are more abundant high-frequency harmonics. The new audio data has good timbre and fluency, sounds clearer, and the tone makes the person seem younger, but it can be completely identified as the voice of the same person.

[0199] According to the mapping combination results of the rich harmonic signals, the spectrogram can be adjusted as needed to obtain the desired timbre and tone effects. A higher tone can improve the hearing effect of hearing-impaired listeners.

[0200] Mapping transformation enables slow playback of polyphonic and harmonious music audio without changing pitch

[0201] Music is an indispensable spiritual work in human daily life, and the processing of music audio is of great significance. Pitch-invariant slow playback is an important branch of music audio processing, and there are still technical challenges. Pitch-invariant slow playback based on waveform similarity can generally process music audio signals with a single sound source well. When the music audio signal has multiple sound sources (multiple voices, harmony), it is difficult to obtain satisfactory results.

[0202] The frequency mapping method independently maps sound signals of different pitches (i.e. different voices and harmonies) in the frequency domain, and can ensure the original smoothness and timbre of all voices and harmonic audio signals when played slowly without changing the pitch.

[0203] Fig.16 Comparison of the slow playback results of polyphonic and harmonious music audio data using the WSOLA method and the frequency mapping method. Fig.16 (a) is the original audio. Fig.16 (b) in the figure is the WSOLA method that plays audio slowly. Fig.16 (c) in the figure is the audio using the frequency mapping method.

[0204] Fig.16 (Top) The picture shows some audio of Sheng Zhongguo playing the second movement of Chen Gang and He Zhanhao's "Butterfly Lovers" violin and cello duet. The violin and cello play two parts with different pitches at the same time. Since the WSOLA method (1 / 4 slow playback) can only effectively process the sound with strong volume at the same time, it cannot take into account the two parts of the violin and cello at the same time. The processing result has a broken fundamental tone and poor fluency.

[0205] The frequency mapping method (1 / 4 slow playback) can accurately reconstruct the sounds of both the violin and cello parts at the same time, completely maintaining the smoothness and timbre of the original audio.

[0206] Fig.16(Middle) The picture shows part of the audio of Barber's "Adagio for Strings" played by a string orchestra. Two very stable parts with different pitches (and a lot of chord sounds) sound at the same time. The WSOLA method (1 / 8 slow playback) cannot accurately reconstruct the sounds of the two parts at the same time. The result is poor fluency, and the audio volume fluctuates and becomes very unstable.

[0207] The frequency mapping method (1 / 8 slow playback) accurately reconstructs the sounds of two parts with different pitches, completely maintaining the original audio timbre, and the smoothness is also very good, and the audio volume remains extremely stable.

[0208] Fig.16 (Below) The picture shows part of the audio of the second movement of Beethoven's piano sonata "Moonlight". The performance runs through two parts and chords (played by both hands at the same time). The pitch of the sounds played by all fingers of the left and right hands is different, so the piano piece is an audio of multiple sounds of different pitches playing at the same time. The WSOLA method (1 / 4 slow playback) has poor fluency and a certain degree of distortion in the sound quality.

[0209] The frequency mapping method (1 / 4 slow playback) accurately reconstructs the sounds of all the different pitches, and the auditory smoothness is very good.

[0210] A brief description of the basic characteristics of EEG signals and the application effect of frequency mapping transformation time-frequency analysis

[0211] Brief description of the basic characteristics of EEG signals

[0212] Electroencephalogram (EEG) is defined as the electrical activity of brain neurons recorded on the surface of the scalp. EEG signals comprehensively reflect the physiological activities of the nervous system and are a powerful tool for analyzing neural activity and brain state.

[0213] EEG classification: spontaneous EEG and evoked EEG.

[0214] Spontaneous EEG is generally divided into 5 frequency bands ( Fig.18 ):

[0215] δ(<4HZ), θ(4-8HZ), α(8-13HZ), β(13-30HZ), γ (>30HZ)

[0216] Rhythmic activity in each frequency band has specific scalp distribution and biological significance.

[0217] Characteristics of EEG signals: very strong randomness and non-stationarity, nonlinearity, and prominent frequency characteristics. EEG signals have rich frequency components, and different frequency components are closely related to brain activities.

[0218] There are multiple sub-components in EEG that overlap in time and are difficult to separate in frequency.

[0219] Improving the time and frequency resolution of EEG signals can more accurately determine the time and frequency position of EEG signals and improve the analysis accuracy of EEG signals.

[0220] Time-frequency spectrum characteristics of EEG signals: The frequency characteristics of EEG change rapidly with time (extremely non-stationary), the EEG frequency components are very rich, and the time-varying characteristics of the time-frequency spectrum convey important information about neurophysiological activities.

[0221] Fig.19 for Fig.18 The time-frequency diagram and spectrum diagram of EEG (average amplitude curve for the entire period). The time window length of the time-frequency analysis is relatively long (1000ms), and the frequency resolution is relatively high, but the time resolution of each rhythmic EEG feature is extremely low, and the time-varying features are fuzzy.

[0222] (2) Application effect of EEG signal frequency mapping transformation time-frequency analysis

[0223] Mapping transformation of spontaneous EEG signals (5 rhythm distributions) Application example 1 - EEG acquisition with eyes closed

[0224] Fig. 20 and Fig.21 The results of applying the mapping transformation to a spontaneous EEG signal collected with eyes closed.

[0225] Fig. 20 The amplitude spectrum is displayed, and the time window length of the time-frequency analysis is 200ms. In the original EEG signal time-frequency diagram and spectrum diagram, the frequency resolution is low, the δ, θ, and α rhythms cannot be well distinguished, and the internal details of each rhythm are unclear.

[0226] Under the premise of the same time resolution (same analysis window length), the frequency mapping transformation processing [c(f)=3f] maps each rhythm component to a position three times the original frequency, which greatly improves the frequency resolution. The EEG characteristics of each rhythm are clear and detailed, and the change trends in the time and frequency directions are clearer.

[0227] Fig.21 Displayed in power spectrum, the time window length of time-frequency analysis is 200ms ( Fig.21 (b) in the figure), 400ms( Fig.21 (c) In the time-frequency spectrum of the original EEG signal, the frequency resolution is low, the boundaries between the rhythms are blurred, the internal changes are unclear, and the delta rhythm (<4Hz) cannot be identified.

[0228] The results of frequency mapping transformation [c(f)=3f] show that the frequency resolution is greatly improved compared with the original EEG in the time-frequency spectrum of different analysis window lengths, the internal changes are clear, the time-frequency information of each rhythm is richer, and the delta rhythm (<12Hz) can be well identified. The frequency mapping transformation provides more accurate and clear basic data for distinguishing the EEG signals of each rhythm in the time and frequency directions.

[0229] Mapping transformation of spontaneous EEG signals (5 rhythm distributions) application example 2

[0230] Fig. 22 The result of applying the mapping transform to a spontaneous EEG signal. The frequency resolution of the original EEG signal spectrum (above) is low, the boundaries between the rhythms are fuzzy, the internal changes are unclear, and the delta rhythm (<4Hz) cannot be identified.

[0231] Frequency mapping [c(f)=3f] The frequency resolution of the EEG signal (below) has been greatly improved, and the time-frequency information of each rhythm is richer. The temporal variation characteristics of the delta rhythm (<12Hz) are very clear and can be well identified.

[0232] Application of Mapping Transformation on Visually Evoked Electroencephalogram (VEP) Signals

[0233] Fig.23 The mapping transformation is applied to the visual evoked EEG signal (VEP). The time window lengths of the time-frequency analysis are 200ms ( Fig.24 (b) in the figure), 400ms( Fig.23 (c) in the figure). The frequency resolution of the original EEG signal is low. When the time window length is 200ms, the VEP is clearly visible, but the P300 component is completely invisible. When the time window length is 400ms, the P300 component is visible, but the duration and position are inaccurate.

[0234] The results of frequency mapping transformation processing [c(f)=3f] show that the frequency resolution is greatly improved relative to the original EEG in the time-frequency spectrum with different analysis window lengths. When the time window length is 200ms and 400ms, the VEP and P300 components are clearly visible and the starting time position is more accurate, and the energy intensity relationship in the time-frequency spectrum is consistent with the time domain signal intensity relationship.

[0235] The frequency mapping transformation results more clearly show the time-frequency characteristics of VEP and P300 components. Application of mapping transformation on laser evoked electroencephalogram (LEP) signals

[0236] Fig.24 To map the results of the application of transformation to laser evoked EEG signals (LEP), the time window lengths of the time-frequency analysis were 200ms ( Fig.24 (b) in the figure), 400ms( Fig.24 (c) in the above.

[0237] Event-related synchronization ERS (event-related synchronization) and event-related desynchronization ERD (event-related desynchronization) are non-phase-locked signals that are difficult to obtain in the time domain. They can be identified in event-related spectral perturbation ERSP (event-related spectral perturbation).

[0238] The frequency resolution of the original EEG signal is low. Fig.24 (b) in the figure), the LEP feature is unclear and the ERS is not visible; when the window length is 400ms ( Fig.24 In (c), LEP is improved.

[0239] The results of frequency mapping transformation [c(f)=3f] show that the frequency resolution of the time spectrum with different analysis window lengths is greatly improved compared with the original EEG. LEP and ERS are clearly visible at the 200ms and 400ms window lengths, and ERD is also improved.

[0240] Application of Mapping Transformation on EP42 EEG Signal

[0241] Fig.23 This is the result of applying the mapping transform to the EP42 EEG signal. The time window length of the time-frequency analysis is 1600ms. In the time-frequency spectrum of the original EEG signal, the frequency resolution is low, the internal changes are not clear, and the frequency direction details of the low-frequency part (<3Hz, white box) cannot be distinguished.

[0242] The frequency resolution of the frequency mapping transformation processing [c(f)=3f] is greatly improved, the time-frequency information is richer, and the frequency resolution of the low-frequency part (<9Hz, white frame) is particularly improved, and three different frequency component signals can be separated.

[0243] Frequency mapping transformation processing is an effective tool for analyzing extremely low frequency EEG signals.

[0244] Application of mapping transformation when EEG signals are played as audio

[0245] Fig.26 This is the result of the mapping transformation when the EEG signal is played as audio. When the original EEG signal is played as audio at normal speed, you can hear dense and short "tick" sounds. Because the "tick" sounds are relatively dense, it is difficult to distinguish the interval between the previous and subsequent "tick" sounds.

[0246] When the mapped EEG signal [c(f)=3f] is played at 1 / 3 the speed of audio, the dense and short "tick" sound becomes slow, and the previous and subsequent "tick" sounds can be clearly distinguished, while the auditory fluency and timbre of the original EEG signal when played at normal speed are completely maintained. As the playback speed is slowed down, the human ear can hear more details and the time resolution is improved.

[0247] Playing EEG signals as audio is an attempt at a new method of EEG signal analysis.

[0248] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method, characterized by: The following steps are involved: Collect audio and EEG signals, and pre-process the input audio and EEG signals; The time series signal is segmented by a sliding time window, and Fourier transform is prepared for the signal segment within each time window; Perform Fourier transform on each short-time window signal, convert the time domain signal into the frequency domain, and obtain the signal spectrum diagram; Design and apply a mapping function to map the spectrum, calculate the phase adjustment amount according to the mapping transformation mode function, and apply it to the mapped spectrum; The process of calculating the phase adjustment amount and applying it to the mapped spectrum is as follows: Assume the time domain signal is , the frequency domain is , the frequency domain mapping is transformed into ; Assume that after mapping transformation The processed time domain signal is , the frequency domain is ,but: ； (1) definition For the mapping mode function, different mapping mode functions are designed according to different expected outputs to obtain different mapping transformation results; Calculate the phase adjustment amount according to the mapping transformation mode function and apply the phase adjustment amount to the mapped spectrum In order to compensate for the phase change introduced in the mapping process, the final mapping result is obtained; The frequency-mapped and phase-adjusted spectrum data is converted back to the time domain through inverse Fourier transform to obtain a signal with high time and frequency resolution; The acquisition process of the signal with high time and frequency resolution is as follows: Prepare frequency mapped and phase adjusted spectral data samples ; After processing, the spectrum data samples Apply an inverse discrete Fourier transform to convert it from the frequency domain back to the time domain; For the signals divided into multiple short-time windows, inverse discrete Fourier transform is performed, and all the short-time window signals after inverse discrete Fourier transform are merged to reconstruct the complete time domain signal; Check the reconstructed signal samples Whether the characteristics of the original signal are maintained, and whether the time and frequency resolution are improved, and the reconstructed signal is further processed by applying a window function to reduce boundary effects and optimize signal quality; According to the time domain signal obtained by inverse Fourier transform, the time-frequency spectrum diagram is redrawn, and the time-varying spectrum characteristic information of the signal is obtained by analyzing the time-frequency spectrum diagram.

2. The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method according to claim 1, characterized in that: The acquisition and preprocessing process of the audio and EEG signals is as follows: Collect audio signals, convert them into digital format, and perform pre-processing operations such as denoising, filtering, and standardization on the audio signals; Perform endpoint detection on the speech signal to determine the start and end points of the speech signal and remove the silent segments; The EEG cap is used to collect EEG signals on the surface of the scalp, which are amplified by an amplifier, and the EEG signals are preprocessed by denoising, filtering, baseline correction, and artifact removal. According to the experimental design and task requirements, the preprocessed EEG signals are divided into different time periods, marked and classified.

3. The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method according to claim 2, characterized in that: The segmentation and Fourier transform process of the time series signal is as follows: Determine the time window length based on the signal characteristics and analysis requirements , the time window length is in the number of samples; For audio signals, the time window length is the number of samples corresponding to 5ms, and for EEG signals, the time window length is the number of samples corresponding to 200ms; The signal is divided into multiple short time window signals through a sliding time window, and each time window has a fixed step size. Move to cover the entire signal, fixed step size Less than the time window length ; Mark each short-time window signal as ,in is the sample index, from 0 to .

4. The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method according to claim 3, characterized in that: The process of acquiring the spectrum of the signal is as follows: For each short-time window signal Apply discrete Fourier transform to obtain its frequency domain, expressed as ; Calculate the absolute value of the amplitude and power spectrum , analyze the energy distribution of the signal at different frequencies; Using the formula Frequency Index Convert to actual frequency value ,in, is the sampling frequency of the signal; By calculating each short-time window signal and , and its corresponding frequency Pairing, constructing the spectrogram of the signal; Repeat the processing of the entire signal to obtain the energy distribution of the signal at different times and frequencies, which is the time-frequency spectrum of the signal.

5. The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method according to claim 4, characterized in that: The calculation formula for obtaining the frequency domain by applying discrete Fourier transform is: ； in, It is The complex amplitude of the frequency component, It is The short-time window signal value of samples, is the time window length, is the frequency index, , Is an imaginary unit.

6. The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method according to claim 5, characterized in that: The processed spectrum data samples Applying the inverse discrete Fourier transform, the calculation formula is: ； in, are the reconstructed signal samples, is the processed spectrum data sample, is the number of transformation points, which is equal to the time window length, is the sample index in the time domain, Is an imaginary unit.

7. The high-resolution audio and EEG signal time-frequency analysis method based on frequency mapping method according to claim 6, characterized in that: The drawing process of the time-frequency spectrum diagram is as follows: The time domain signal obtained by inverse discrete Fourier transform is analyzed again in time-frequency direction, the signal is segmented in time direction using sliding window technology, and the signal samples reconstructed in each time window are Perform Fourier transform to obtain the spectrum of each time window ,in Indicates the sequence number of the time window; The spectrum of each time window is calculated by taking the square of the spectrum modulus The power spectral density of , where the square of the spectrum modulus is calculated as: ; Indicates that in the time window and frequency index The power spectral density at ; Arrange the power spectrum density in each short time window in time order to form a time-frequency spectrum diagram; Draw a time-frequency spectrum to visualize it. In the time-frequency spectrum, the frequency On the vertical axis, the time window On the horizontal axis, the power spectral density Indicated by color; Observe the time-frequency spectrum diagram, analyze the changes of the signal at different times and frequencies, and through the time-frequency spectrum diagram, intuitively see the trend of the signal's spectral components changing over time, thereby obtaining the signal's time-varying spectral feature information.

Images